Speaker John Bercow and two Labour MPs rounded on the man charged with administering benefits to thousands of disabled people for what were described as "offensive and discriminatory" remarks on Wednesday.
They were responding to an observation made by the Tory minister on Monday, who, while responding to news of a rise in the number of people with disabilities returning to work, commented that it was important the figure rise further to meet the employment rate of "normal, non-disabled people".
A spokesperson for Duncan Smith refused to comment at the time, but his remark was quickly criticised as "shocking" and "shameful".
The issue was raised at Prime Ministers Questions by Labour MP Debbie Abrahams.
Disabled people have been hit by benefit cuts in recent years
She called for an investigation into whether the work and pensions secretary had broken the ministerial code.
Her intervention was swiftly followed up by a point of order from colleague Andy McDonald, who blasted Duncan Smith's language as “offensive and discriminatory”.
Bercow, mediating proceeding as House of Commons Speaker, responded by imploring all MPs to consider the impact of their words.
“The language used was not disorderly, there was nothing out of order about it and therefore it didn’t necessitate any intervention from the chair. Everybody who speaks in this place must take responsibility for what he or she says," he told parliamentarians.
But, he added, “It is extremely important that we express ourselves with sensitivity and in a way which will be viewed as such, not just within the House but beyond it.
"I don’t think that I can go beyond that today. I recognise the upset that the honourable gentleman feels and, perhaps, that other feels.
“I don’t think that I can say more than what I’ve just said, but I am sure that what he has said and what I’ve said in response will be noted in the appropriate quarters and I hope it won’t be necessary to return to this theme.”
Wednesday's intervention came after SNP MP Dr Eilidh Whiteford told The Huffington Post UK that Duncan Smith's comment was just the latest case in a long line of attempts to vilify those with serious health afflictions.
IDS made his original remarks on Monday
"I thought it was quite a revealing use of language," she said.
"Given what we know about the very high proportion of those who work with disabilities, to say they're abnormal is not right; it creates a 'them and us' kind of attitude.
"I was quite taken aback," Dr Whiteford added. "I thought the discourse was more evolved than that.
"This government has spent the last five years vilifying disabled people as work-shy, despite the fact some of them have very serious health problems."
- Group Of Student Lawyers Overturn 95% Of DWP Decisions
- UN To Investigate DWP Over Disabled People's Human Rights Abuses
- Iain Duncan Smith Should Resign, Says Jeremy Corbyn