Pregnant women have been advised to avoid alcohol altogether and warned that drinking even one or two glasses of wine a week during pregnancy can lower their child's IQ.
Researchers looked at the IQ scores of 4,000 children as well as recording the alcohol intake of their mothers.
They found 'moderate' alcohol intake of one to six units a week during pregnancy affected IQ. They said the effect was small, but reinforced the need to avoid alcohol in pregnancy.
Dr Ron Gray, from Oxford University, who led the research said lower IQ had been shown to be associated with being socially disadvantaged, having poorer health and even dying younger.
"It is for individual women to decide whether or not to drink during pregnancy, we just want to provide the evidence," he said.
"But I would recommend avoiding alcohol. Why take the risk?"
The research, published in the PLOS One journal, looked at changes in the genes that are not connected to social or lifestyle effects.
The study found that four genetic variants in alcohol-metabolising genes in children and their mothers were strongly related to lower IQ at age eight.
On average, the child's IQ was almost two points lower per genetic modification they possessed. But this effect was only seen among the children of women who drank between one and six drinks a week during pregnancy and not among women who abstained when they were pregnant.
The researchers said the study strongly suggested that it was exposure to alcohol in the womb that was responsible for the differences in child IQ.
Since 2007 the Department of Health's advice has been that women who are trying to conceive or are pregnant should avoid alcohol.
But Dr Clare Tower, consultant in obstetrics and fetal maternal medicine, at St Mary's Hospital, Manchester, stressed to the BBC that women who have had the occasional alcoholic drink in pregnancy should not be overly alarmed by the findings.
"Current UK advice is that the safest course of action is abstinence during pregnancy," she said.
"The finding of this study would concur that this is undoubtedly the safest advice."
But she said that another recent study had found no effect on IQ at five years.
"It is likely therefore, that any impact is likely small and not seen in all women," she added.
Did you have occasional glasses of wine during pregnancy?
More on Parentdish: Drinking alcohol in pregnancy. What are the risks?