Patten singled out leaving the EU among a series of factors – citing its impact on the pound, imports and labour costs – as he argued Lithuania and Poland are powering ahead.
He began: “The truth is we’re in one hell of a mess. Our GDP per capita now is less than not only France, Germany, the Netherlands, it’s lower than Ireland.
“It’s lower for heaven’s sake than Lithuania. The poorest 20% in Britain are poorer than the poorest 20% in Poland. That is not the sign of a country which has things going for it.”
Patten went on to point to comments made by Larry Summers, the former US treasury secretary, who said Britain is in a much worse position than others.
Patten did mention factors including inflation caused by government spending money “which was necessary” to deal with energy prices and furlough, as well as “lax” monetary policy by the Bank of England.
He added: “But it’s also, and this is a word one isn’t supposed to use anymore, because of Brexit.
“It’s because of what Brexit did to the value of the pound. It’s because of what Brexit has done to make it more difficult for us to import goods and for us to import labour and importing food.
“Now it’s costing us, according to the (London School of Economics), £7 billion more a year because we’re outside the European Union.
“And until we start facing up to the realities of our life, to the fact that we can’t possibly spend money on all the things we want, increase public spending and cut taxes, it’s absolutely impossible.”
He continued: “I think Rishi Sunak and Keir Starmer are decent human beings ... I just wish they would fess up to the fact that we have to tell people what they’re going to have to expect in the next few years. I think it’s going to be very tough to get out of the hole that we’re in.”
Also on the show, food campaigner Jack Monroe hit out at the labelling of the cost-of-living crisis – arguing the squeeze on household budgets has been happening for more than a decade and not simply “fallen out of a clear blue sky”.
The popular budget chef, appearing on BBC Question Time on Thursday, argued the problems stem from Conservative austerity and 13 years of “pulverizing” public services that “propped up the fundamentals of a decent society”.