The documentary sparked outrage due to its depiction of benefit claimants in Birmingham, with critics branding it "medieval stocks updated for a modern format" and a "misrepresentation" of those living on social security.
The work and pensions secretary cited the documentary as an example of "ghettoised communities" that are like a "different country" for a speech he made in January.
But despite the secretary of state seizing on the show to justify his changes to the benefits system, the Huffington Post UK can reveal that he declined to meet the residents of James Turner Street.
Residents invited Duncan Smith and the main three party leaders to meet the residents, including those who appeared on the programme, claiming that the show fed a "stigmatisation" of benefit claimants.
But the work and pensions secretary said he could not find a time to make a visit, as shown in correspondence released after a Freedom of Information request. Sources close to Duncan Smith told HuffPostUK that they were "not aware" of any upcoming plans to meet the residents.
Duncan Smith wrote: "We appreciate that the vast majority of people who can work want to work and so our strategy is to help them by making work pay and by improving employment opportunities. Of course, there are those who, for various reasons, may not be in a position to work.
"These can include people with children, the sick and disabled, and also those who are making every effort to find work but who are, through no fault of their own, unsuccessful. This is why assistance from the social security system has been designed with them , and their needs, in mind.
"Unfortunately, due to existing diary commitments, parliamentary and constituency business, I am unable to accept your invitation at this time."
The work and pensions secretary praised the Birmingham Empowerment Forum, the local group which passed on the James Turner Street residents' invitation, for the "valuable work" they do in the community.
Benefits Street's popularity has led to Channel 4 producers confirming that they were working on a new series, thought to be called Immigration Street.
Australia's opposition leader Bill Shorten admitted that he feared that the show would spawn a spin-off down under.
The Labor party leader made his prediction as he spoke at a conference organised by the IPPR and Policy Network think tanks about the welfare system in Australia.
"I understand you've got a television show here called Benefits Street," he told the audience. "Unfortunately, I suspect that'll be copied in Australia but I think that's an unhealthy development."