Those consuming more than 36 grams of alcohol per day - equivalent to a pint-and-a-half of medium-strength beer - experience noticeable declines in memory and problem-solving skills up to six years earlier than light or moderate drinkers, scientists found.
No differences in memory and executive function, which includes reasoning, planning and problem-solving, were seen between non-drinkers, former drinkers and light or moderate drinkers.
The study involved 7,153 Whitehall civil servants, including more than 5,000 men, whose drinking habits were assessed three times over a period of 10 years.
Dr Severine Sabia, from University College London, who led the research reported in the journal Neurology, said: "Much of the research evidence about drinking and a relationship to memory and executive function is based on older populations.
"Our study focused on middle-aged participants and suggests that heavy drinking is associated with faster decline in all areas of cognitive function in men."
Light or moderate drinkers were defined as those who consumed less than 20 grams of alcohol per day.
Dr Simon Ridley, from the charity Alzheimer's Research UK, said: "These latest results could serve as one more reason to stick to any New Year's resolutions to cut back on alcohol.
"Observational studies such as this can be important for identifying factors that may influence the risk of memory decline or disease, but it's difficult to pinpoint cause and effect with this type of research. The people in this study did not have dementia, but memory decline can be a precursor to dementia and understanding the risk factors for this decline could be important for preventing the condition.
"This large study has yielded useful results, although it's not clear how far the findings from this group of civil servants can be generalised to the wider population."