They also underwent brain scans and tests of memory and thinking ability.
An increase in blood pressure after the age of 50 was linked to a greater risk of small brain bleeds, known to be associated with dementia.
At the same time, a reduction in resting or "diastolic" blood pressure between middle and older age was linked to smaller brain volumes and reduced mental performance.
"Our findings bring new insight into the relationship between a history of high blood pressure, blood pressure in old age, the effects of blood pressure on brain structure, and memory and thinking," said lead researcher Dr Lenore Launer, from the National Institute on Ageing in Bethesda, US.
"Older people without a history of high blood pressure but who currently have high blood pressure are at an increased risk for brain lesions, suggesting that lowering of blood pressure in these participants might be beneficial.
"On the other hand, older people with a history of high blood pressure but who currently have lower blood pressure might have more extensive organ damage and are at risk of brain shrinkage and memory and thinking problems."
The findings are reported in the latest edition of the journal Neurology.
Dr Laura Phipps, from the charity Alzheimer's Research UK, said: "Research suggests that lifestyle factors, including blood pressure, could be important modifiers of brain health. This study did not focus on dementia, but it suggests a complex relationship between mid and late life blood pressure, and memory performance later in life.
"As the study did not keep a continuous measurement of blood pressure between midlife and later life, it is hard to know exactly when and why changes in the brain may have occurred. It would be good to see more detailed investigations into how changes in blood pressure throughout life may impact on brain health.
"Evidence suggests that lifestyle factors such as a balanced diet, healthy weight, regular exercise and not smoking, may also help to keep our brains healthy as we get older. If you are concerned about your blood pressure, you should talk to your GP."