David Cameron and his cabinet are increasingly viewed by working and middle classes as out of touch "toffs" who do not understand ordinary people's concerns, a senior Tory MP has warned.

David Davis, who stood against Cameron for the leadership of the party in 2005, said the impact could as damaging to the party as the allegations of sleaze were in the late 1990s.

"The trouble today is, that with so much austerity, so much pressure, a lot of the working class and lower-middle class voters feel that they have been sort of pushed off the escalator and the effect of that is they start to resent those who they think are still on the escalator, or possibly even at the top of the escalator," he said.

"That works worse against Tory politicians than it does against any others ... partly because they think we are better off, they think we are toffs."

The MP for Haltemprice and Howden told the BBC's World at One programme on Thursday that Downing Street need to get a grip on the problem.

Recent furors including the so-called "pasty tax", the decision to scrap the 50p top rate of tax and the bizarre tale of the prime minister going horse riding with his wealthy friends in Oxfordshire may have contributed to an impression that ministers are part of a rich elite.

"The truth of the matter is they look at the frontbench, they see them all very well dressed, well turned out, well fed and perhaps feel that they are in a different world to them," Davis said.

He added: "The trouble is, if it sticks, it's dangerous and difficult to get rid of.

"Governments always have to watch this, that they don't get stuck with an impression that plays against them.

"In John Major's day it was sleaze, if you remember, today this may be its equivalent. If it sticks it makes almost everything you do impossible to make work.

"So I would worry about it. If I were advising in Downing Street I would worry about this. It's not an insuperable problem yet, but it's one they've got to get a grip on and solve."

The comments are not the first time Davis, who was raised on a council estate in South London, has attacked the direction of the government. In February he launched a scathing attack on what he called the “crony capitalism” of ministers, accusing them of being “far too close” to big business.

Labour surged into a 10-point lead following last week's Budget according to one poll. A ComRes survey put Labour on 43%, with the Tories down four at 33% and the Liberal Democrats down two on 11%.

Davis' criticisms were echoed by Tory backbencher Mark Pritchard, who suggested the current Cabinet lacked empathy with voters.

"I think if we had a more empathetic government then perhaps we wouldn't be seeing some of the issues we have seen over the last few days," he said.

Pritchard, a frequent critic of the coalition who serves as secretary of the influential 1922 committee of backbench Tory MPs, said the prime minister should reshuffle his top team to bring in more people from poorer backgrounds.

"I think that those people that perhaps have struggled at some point in their life to pay the electric bill or have perhaps arrived in school in a Morris Minor rather than a Jaguar are more likely to have empathy with those people who are currently struggling out there in the country than perhaps others who haven't had that experience," he said.