One in eight also worry that exercise could bring on premature labour, according to the survey of 1000 pregnant women and new mums commissioned by baby charity Tommy's.
But are they right to be worried?
Both these fears are unfounded and Tommy's believes that myths like these are preventing women from doing enough physical activity during pregnancy.
Tommy's Pregnancy Line midwife Jules Robertson said: "Whatever your fitness level, in the vast majority of pregnancies exercise is helpful not just for you but also for your baby."
Pregnant women are advised to do similar amounts of exercise to everyone else – a minimum of 30 minutes four times a week. The benefits of doing so can include less back ache, breathlessness, tiredness and swelling of the hands and feet. Exercise also improves blood flow to the placenta, which is good for the growth and development of the unborn baby. Staying active also reduces the risk of complications at birth by building the fitness and endurance needed during labour. A study in 2000 found that women who are inactive during pregnancy are 4.5 times more likely than average to need a caesarean section due to problems during labour.
So what should you do?
"Any activity that makes you warm and slightly out of breath," Jules advises. "Brisk walking, gardening, and vigorous housework all count.
"If you didn't do much exercise before you fell pregnant, then build up slowly, for example by doing a brisk walk for 15 minutes three times a week. If you are used to exercise you can carry on doing more vigorous exercise from going to a class to jogging and swimming, adapting or reducing your exercise as your bump gets bigger.
"There are few absolute no-nos apart from scuba diving, but as a rule of thumb avoid anything in a hot environment such as Bikram Yoga, or anything where you might receive an impact to the stomach," Jules adds.
"Becoming a mum is the workout of a lifetime and the added bonus of improved fitness is that you'll have the stamina for all the running around that lies ahead, as well as finding it easier to get back into shape."
Women with complicated pregnancies, such as those with a low-lying placenta or a high BMI will be given different advice tailored to their circumstances by their midwives. Always speak to your midwife or GP before taking up a new form of exercise.
To find out more about the benefits of exercise and what you can and can't do during pregnancy watch the video above from Tommy's and Dr Hilary Jones.
More on Parentdish