Speaking to the Labour Party conference this afternoon, Balls told Labour members of his surprise when his wife, shadow home secretary Yvette Cooper, said Cameron looked "slim" when he was photographed holidaying in Cornwall.
Referring to the photo of the PM papped changing while wearing a Mickey Mouse towel, Balls quipped: "I just thought for a Prime Minister, it was a surprisingly small towel."
A Tory source told HuffPostUK the quip was "puerile and immature", adding: "When that's all Mr Balls has to say, we know it's because he hasn't got any policies and can't be trusted on the economy."
Balls told his audience: "Didn't you feel a little sorry for our Prime Minister this summer? Didn't you?
"Back in August, on the beach, changing into his swimming trunks, behind that Mickey Mouse towel, captured on cameras, unflattering pictures spread across the national press.
"I've been there, I know what it's like.
"Let me let you into a little secret. When Yvette saw the pictures she said, rather pointedly I thought, she said that for a 46-year old man David Cameron looked rather slim.
"Slim? Who on Earth can she have been comparing him to?
Balls also used his speech to members in order to flesh out Labour's economy policy, promising to let the Office for Budget Responsibility have a look at what his party have to offer.