Overdosing on the mineral selenium by taking supplements raised the chances of developing high-grade cancer by 91%, scientists found.
Vitamin E pills also boosted the risk of aggressive cancer, more than doubling it for men lacking selenium.
The researchers believe selenium can turn toxic when present in the body at excessively high levels.
At the same time, the mineral appeared to protect against the harmful effects of too much vitamin E.
The US study was a follow-up of Select, the Selenium and Vitamin E Cancer Prevention Trial, which originally recruited more than 35,000 men to see if the supplements could help prevent prostate cancer.
Researchers stopped the trial three years early in 2008 after there were hints that instead of protecting men, vitamin E was putting them at greater risk, while selenium showed no benefit.
Story continues below the slideshow:
A subsequent comparison of 1,739 participants diagnosed with prostate cancer and 3,117 matched cancer-free individuals highlighted the supplement hazards.
Study leader Dr Alan Kristal, from the Fred Hutchinson Cancer Research Centre in Seattle, said: "These supplements are popular - especially vitamin E - although so far no large, well-designed and well-conducted study has shown any benefits for preventing major chronic disease.
"Men using these supplements should stop, period. Neither selenium nor vitamin E supplementation confers any known benefits, only risks."
Selenium supplements had no effect on men who started out lacking the mineral, but were harmful when added to baseline levels that were already high. For aggressive, high-grade cancers, the risk went up by 91%.
Among men with low selenium status at the start of the study, vitamin E supplements increased the overall risk of prostate cancer by 63% while the high-grade risk rose by 111%.
Of the men in the study who developed prostate cancer, 489 were diagnosed with high-grade disease.
Dr Kristal said people were often misled by the supposed benefits of dietary supplements.
"Many people think that dietary supplements are helpful or at the least innocuous," he said.
"This is not true. We know from several other studies that some high-dose dietary supplements - that is, supplements that provide far more than the daily recommended intakes of micronutrients - increase cancer risk.
"We knew this based on randomised, controlled, double-blinded studies for folate and beta carotene, and now we know it for vitamin E and selenium."
The new findings appear in the latest edition of the Journal of the National Cancer Institute.
In their paper the scientists warn that "men aged greater than 55 should avoid supplementation with either vitamin E or selenium at doses that exceed recommended dietary intakes".
Men taking part in the research were given 400 international units (IU) of vitamin E a day and a selenium dose of 200 micrograms.