The charity claims tax dodgers are costing the UK nearly £200 per household at a time when many are struggling to make ends meet and 220,000 people have turned to charity foodbanks.
Oxfam claims the unpaid tax could go a long way towards helping "rescue millions from the poverty trap".
Oxfam argues that tax avoidance must be clamped down on, and money used to help people in poverty, like those using food banks
Its research, which comes on deadline day for UK tax returns, found money owed by wealthy tax evaders would provide £21 a week to every household in the UK that is experiencing fuel poverty or prevent more than nine million households from being £180 a year worse off by plans to put a 1% cap on welfare uprating to be scrapped.
Alternatively, the cash would double the amount of universal childcare entitlement to 25 hours per week to giving struggling families more flexibility to work, the charity said.
Chris Johnes, Oxfam's director of UK poverty, said: "As many people in the UK submit their tax returns today, billions of pounds owed to the Government are languishing in offshore bank accounts, leaving ordinary people to pick up the tab for support and services that we all rely on.
"It is sickening and immoral that tax evaders get off scot-free whilst thousands of the poorest families are being forced to go to charity foodbanks in order to provide a meal, or go without heating when it's freezing outside."
The research is part of the "Enough Food For Everyone" Campaign, run by a coalition of charities and faith groups, which Oxfam is a part of.
Last week, Prime Minister David Cameron told the World Economic Forum Britain will use its year-long presidency of the G8 group of rich nations to push for global action against tax evasion and "aggressive" tax avoidance by wealthy individuals and businesses.
But Oxfam said the Government is not doing enough and criticised the target set for HM Revenue and Customs' Affluent Compliance Team of clawing back £586 million in unpaid tax by the end of 2015.
Chris Johnes added: "The Prime Minister's promise to clamp down on tax evasion is welcome, and we need to see him act on this. The negative response from the FTSE 100 shows what he is up against- but the vast majority of ordinary people want to see radical action to rein in tax dodging - and now.
"The current targets are far too low. Six hundred million pounds is a drop in the ocean, compared to the billions that are actually owed.
"We must get back all of this lost revenue, not only to help fix our economy, but to reverse cuts to support that the most vulnerable need and stop the shame of poor people in Britain going hungry in the 21st Century."
Responding to the Oxfam report, a Treasury spokesman said: "We do not recognise this figure but while the majority of people and businesses pay what they owe, the Government is fully committed to clamping down on the minority who don't.
"This is why this Government has invested an extra £1 billion in HMRC to tackle evasion, avoidance and fraud which will bring in £22 billion a year by 2015.
"Last year, HMRC brought in £16.6 billion due to compliance activity alone and just yesterday we received £342 million from Swiss bank account holders.
"This deal is expected to bring in over £5 billion over the next six years and is the largest tax evasion settlement in UK history."