The Tory minister said there were many reasons why people relied on them and that insisting it was just to do with benefits was "quite wrong".
This is despite 913,138 people using food banks in 2013-14 according to the Trussell Trust, a figure which the charity says is the 'tip of the iceberg' of the extent of poverty in the UK.
His comments come after the Church of England-backed Feeding Britain report identified delays in welfare payments and sanctioning of claimants as key factors driving disadvantaged households to food banks.
One million food parcels from #foodbanks last year and IDS says the use is "tiny" Words fail me— Mark Russell (@markrusselluk) December 14, 2014
Duncan Smith told the BBC's Sunday Politics that people going back to work were less likely to want to use food banks.
He added: "In Germany, food banks are used every week by 1.5 million people. It is tiny in proportion here compared to a place like Germany which has more generous benefits and in which you have a higher level of pay.
"So just saying it is to do with benefits is quite wrong.
@David_Cameron but lengthened the food Q-do u agree with IDS-even well off use food banks? a vile remark insult to poor u create— bachulus (@bachulus) December 15, 2014
"What I do say is there are lots of other reasons lots of people go to food banks."
He also defended the speed with which benefits are processed, saying that when the Coalition came into power only 88-89% of payments were on time compared to 96-97% now.
But he accepted that the advanced payment all job seekers are entitled to needed to be better publicised.
Well if the germans jumped in a river sbould we do the same??
He should be proud that in 2014
we have food banks, Shame on u IDS.— Pat Shewbridge (@1994timothy) December 15, 2014
He said: "If they have a difficulty financially, there is money available to tide them over any particular area or time they have a problem with.
"Available to every single person in a job centre who is making a claim is, say they have an issue or a problem, they can get an advanced payment. Now what the report said which I agreed with was that we need to do more to advertise that. So we are doing that now and I accepted that."
Duncan-Smith also said limiting child benefit to the first two in a family was "one of the proposals well worth looking at" and added: "It helps behavioural change."
On Personal Independence Payments he accepted the early phases did not go "exactly according to plan", but insisted: "All of those delays will be back on track by end of year."
He said Employment Support Allowance was in need of reform, but pointed out the appeals level had fallen to "hardly any at all".
And housing benefit fell in real terms for the first time last year, he added, after the Government inherited a programme "out of control".
Former Archbishop of Canterbury George Carey urged people to mark Christmas by providing food to those unable to afford to feed themselves properly over the festive period.
"Christmas is a time to rejoice and celebrate, to share food together," he told the Murnaghan show on Sky News.
"We ought to share food with the more vulnerable, the needy people, help them out of their distress.
"This is an opportunity for us to be more neighbourly, to give food to others, why not make a contribution to your local food bank for example, all kind of things we can do to bring out the spirit of Christmas and to care for one another and hopefully go into the new year with a sense of optimism for our country, that we are a great country and to put the greatness back into Britain once more."
While people were not "starving to death" in the UK as they were in Africa - the country was returning to the sort of levels of hardship experienced before the creation of the NHS, he suggested.
"Parents are giving up food in order to allow their own children to go without starvation.
"I think that's a terrible thing for our country to fall into this kind of trap so we have to, I think, do something and Christmas is a really timely point of the year to reflect on our priorities."