In the poll of 2,500 men, almost half (48%) said they desperately want to lose weight while 41% want to tone up.
Being the butt of their mate’s jokes, getting undressed in front of other men and feeling the need to live up to super-fit celebrities and models were all found to contribute to men’s overall lack of body confidence.
Being compared to an ex-partner. or even a sibling, was also found to make men doubt their looks.
The research was commissioned by menswear retailer Jacamo, which has since teamed up with cricket legend Freddie Flintoff and comedian Jack Whitehall to help men tackle their body image issues.
Speaking of his own body image struggles towards the end of his career, Flintoff says: "You’re 37 with your belly hanging over your trousers and you’re surrounded by fit lads in their early twenties, you feel old."
According to the report, a major complaint shared by those suffering a lack of body confidence is watching films or television programmes that they consider to feature unrealistically attractive men with good bodies (16%).
Around one in 16 (7%) said this pressure is made worse when their partner looks at images of men with muscular bodies they don’t feel they can ever achieve.
When it comes to key problem areas, more than half of men (54%) dislike their stomachs, while one in seven (14%) wish they had bigger pecs.
A total of 18% of those involved in the survey said they’d like to buy more fashionable clothes, but find clothes shops intimidating.
To open up discussion about male body image and help men deal with their concerns, Jacamo has created a film with Freddie Flintoff and Jack Whitehall where the pair share details of their own struggles.
After Flintoff says being surrounded by younger players made him feel "old", Whitehall confesses he suffers from heartburn during intimate situations and keeps a bottle of Gaviscon by his bed.
He jokes: "She knows as soon as that cap comes off, its business time."
Stuart Donaldson MP, vice-chair of the all-party parliamentary group on body image commented: "The report reveals that anxiety about body image is an issue for men just as it is for women.
"It’s refreshing in its approach about raising awareness of the issue of male body confidence, a subject that many men may find it difficult to talk about."