Three out of four people think bosses at bailed-out banks should not get a bonus, according to research.
And 58% of respondents to the YouGov poll, commissioned by Sky News, said Britain's business reputation is being damaged by the actions of bankers.
Sir Philip Hampton, chairman of the Royal Bank of Scotland (RBS), said bonuses cannot continue at the current level.
"Part of the reason for the pay is that the profits were not sustainable.
"They were there for a few years but they were not sustainable and the pay moved up to that level of profits and it now needs to be corrected down," he told Sky News.
Sir Philip turned down a £1.4 million bonus earlier this month.
But Nigel Rudd, former deputy chairman of Barclays, claimed he would have paid more money to Barclays' former chief executive John Varley.
He said: "Bob Diamond (current chief executive) and John Varley made a huge difference to Barclays as they went through this terrible period.
"You realise Barclays never made a loss throughout all this period? I think John Varley was underpaid actually ... because I think what he did throughout that crisis was phenomenal."
The comments come after weeks of conflict over bankers' bonuses, in which RBS chief Stephen Hester turned down his £963,000 bonus amid mounting pressure and Lloyds boss Antonio Horta-Osorio waived his payout following a leave of absence.