Speaking on Tuesday night, Duncan Smith said he was "not in a position" to comment on hitting high-value properties with a mansion tax, which Labour has suggested implementing on houses worth more than £2 million.
"If I started talking about taxes on buildings, I'd have Eric Pickles sitting on me, and that'd be unpleasant," he quipped.
The work and pensions secretary made the jibe at a fringe event hosted by the Centre for Social Justice at the Conservatives' last party conference before the general election in Birmingham after being asked by Tory commentator Tim Montgomerie about whether more taxes should be levied on the wealthy.
Duncan Smith is not the first Tory cabinet minister to poke fun at Pickles' weight; George Osborne did likewise last year.
"My Rt Hon friend the communities secretary has set an example to all his colleagues by reducing the size of his department by 12% and abolishing 12 quangos," the Chancellor told MPs, adding to cheers all around: "He is the model of lean government."
David Cameron has also indulged, previously calling Pickles "the big man on the side of the people" and joking about him running.
Duncan Smith did not just joke about Pickles in his answer tonight, telling Montgomerie that he had been consistently arguing for "balance and fairness" in how the coalition cuts were implemented.
This comes after Osborne unveiled how the Tories would make an estimated £3 billion out of a planned £25 billion of cuts by freezing welfare for working Britons - a move that was condemned as targeting "the poor and the vulnerable".
Duncan Smith insisted: "I believe that by the time we get to the next election that I will be able to stand on a platform where I am able to say that we have not balanced the books on the backs of the poorest."
"Even now people who are earning more are paying more than they ever did in the lifetime of the last government. It's about what you do with the people who are the wealth creators, who innovate, who settle here create jobs and give jobs to people."