The basic rate of tax could be slashed by at least 2p if people paid more of what they owe, Danny Alexander has said.
The Treasury chief secretary said everyone else had to pick up the tab when the wealthy avoided their obligations.
But he also delivered a thinly-veiled rebuke to David Cameron for wading into the row over comedian Jimmy Carr's use of a tax avoidance scheme.
The senior Liberal Democrat's comments came in an interview with the BBC's Sunday Politics.
Asked about the prime minister's attack on Carr, Mr Alexander said: "As a Treasury minister I would not get into the details of any individual's tax affairs. We have taxpayer confidentiality in this country.
"But what is very important, and what I very strongly agree with the prime minister about... is that the kind of aggressive tax avoidance that has been well reported over the course of the week, the government has been dealing with since we came to office, is something that has to be cracked down on."
He went on: "We have to make sure that everybody, especially the rich and famous, are paying their fair share of tax.
"These sorts of schemes that save wealthy people potentially tens of millions of pounds in tax, they are paid for by everybody else.
"If we could narrow the tax gap in this country by a quarter we could reduce income tax for every basic rate payer by 2p in the pound."
Former prime minister Tony Blair also refused to criticise individuals. But he suggested the public were only willing to tolerate such avoidance in the good times.
"I think the mood on this has changed," he told the BBC's Andrew Marr show. "What people maybe would not have cared about a few years back, if you are in a time of austerity people they care about these things."