Iain Duncan Smith thinks zero-hours contracts aren't that bad, they just have an image problem.
He told the BBC: "It should be flexible hours contracts, but only 2% of the total workforce have those and they are mostly people like carers, who can't give direct time, and young people like students, so for them there is a reason for those."
Labour and trade union leaders criticised the comments, saying it showed the minister was "out of touch".
The Opposition has pledged to ban exploitative zero-hours contracts, while unions have been pressing for them to be scrapped altogether.
Under the controversial contracts, workers are often told at the end of the week how many hours they can work the following week.
Launching Labour's manifesto for young people in Lincoln, party leader Ed Miliband said: "Young people are three times more likely to be on zero-hours contracts than the rest of the population.
"Iain Duncan Smith has tried today to rename zero-hours contracts as 'flexible contracts'. We don't need to rename zero-hours contracts, we need to ban exploitative zero-hours contracts, and that's what the next Labour government will do."
Rachel Reeves, shadow work and pensions secretary, added: "Iain Duncan Smith's comment shows again how completely out of touch David Cameron's Tories are with the lives of working people. We don't need to rename exploitative zero-hours contracts, we need to ban them.
"For too many people zero-hours contracts leave people without a regular income and not knowing from one day to the next how much work will be coming in. It's insecurity dressed up as flexibility.
"David Cameron said he couldn't live on a zero-hours contract but the Tories are happy for working people to be stuck in low paid zero-hour contract jobs."
Len McCluskey, leader of the Unite union, said zero-hours contracts meant "misery" or workers and their families, leaving them on a "hand to mouth" existence.
"This insecurity has exploded on David Cameron's watch, where we have an economy built on shaky jobs and chronic low pay."
TUC general secretary Frances O'Grady said: "Whatever you call a zero-hours contract, it is still a recipe for insecurity and exploitation, where the boss holds all the power.
"Nothing sums up what is wrong with this Government's approach to work and how out of touch they are with the way most people live their lives than this celebration of the zero-hours contract.
"If you put lipstick on a pig, it is still a pig. But as 'zero-hours contracts' is a bit of a mouthful, perhaps we could call them 'IDS jobs' in the future."
Duncan Smith hit back at his critics, pointing to a survey carried out by personnel managers which found people on zero hours contracts scored higher on the work life balance than those on fixed hours.
"I am furious with the Labour Party. I am genuinely furious with what they have tried to do about this. They've said 'Oh, there's an epidemic of people' - just over 2% of the total workforce is actually in what they call zero hours contracts," he told BBC Radio 4's The World at One.
"These contracts are for the most part contracts taken by people who have caring responsibilities, students, who might be able to work 20 hours one week, the next week, because you're taking somebody to hospital, you can only work 10 hours."
For Labour, shadow business secretary Chuka Umunna said Duncan Smith had his "head in the sand".
"People on exploitative zero hours contracts are furious with him and that's before you get over their outrage at his introduction of things like the bedroom tax," he told The World at One.
"I just think he is living in a parallel universe. If zero hours contracts are not a problem why did the Prime minister the other day say that he wouldn't want to work on one.
"Instead of tackling exploitation, here you have a man denying the problem and trying to dress up insecurity as flexibility."