More than 2,100 women aged 65 to 80 were given annual tests of thinking and memory skills by researchers from the University of Iowa over six years.
The results revealed that there was "no difference" in cognitive ability between women with high and low levels of omega-3, the study's author Eric Ammann said.
There was also no change in how quickly their thinking skills declined over time, the research found.
Earlier studies had indicated that omega-3s found in nuts and in fatty fish such as salmon may benefit thinking skills.
Mr Ammann, from the University of Iowa, said: "There has been a lot of interest in omega-3s as a way to prevent or delay cognitive decline, but unfortunately our study did not find a protective effect in older women.
"However, we do not recommend that people change their diet based on these results.
"Researchers continue to study the relationship between omega-3s and the health of the heart, blood vessels, and brain.
"We know that fish and nuts can be healthy alternatives to red meat and full-fat dairy products, which are high in saturated fats."
A spokesman for the Alzheimer's Society said the research was "inconclusive".
"This isn't the first study to suggest that omega-3 oil does not protect against cognitive decline," he said.
"It's important to note that the study looked at cognitive decline due to ageing and not specifically at dementia, which is caused by diseases of the brain.
"Don't avoid your favourite fish supper or a handful of cashews on account of this research.
"The best thing people can do to try and reduce their risk of developing dementia is to eat a healthy varied diet and take regular exercise."
The study's findings are published in Neurology online journal.