Quitting the European customs union risks causing a 4.5% fall in GDP by 2030, Cabinet ministers have reportedly been warned.
Pulling out of the agreement could also clog up trade through Britain's ports, economic reports circulated at a Brexit Cabinet committee said, according to the Guardian.
The forecast, based on a Norway-style model for Britain's future relationship with the European Union, is the average prediction made in three studies carried out ahead of the referendum, the newspaper said.
Liberal Democrats said it showed that supporters of a so-called hard Brexit were treating the economy as a "political plaything".
Leader Tim Farron said: "This analysis shows that for all the bluster from Liam Fox and others, the hard Brexit they support will have real and damaging consequences for our economy and every household in this country.
"We now have ministers at the very top of government who refuse to listen to evidence, to business and to economic fact. They are treating our country's economy as there own political plaything. They are not worthy to be in government."
The customs union covers all EU states, along with Turkey and a number of smaller countries like Monaco, Andorra and San Marino and non-EU UK territories such as the Isle of Man, Jersey and Guernsey.
Members can trade goods without customs levies, but must impose common tariffs on goods from outside the area and - crucially - negotiate trade deals as part of the union and not as individual states.
The warnings come as Theresa May prepares to head to Brussels on Thursday - 100 days since taking office - to update European counterparts on Britain's plans to quit the EU.
The symbolic anniversary will see the Prime Minister address the rest of the bloc informally on the "the current state of affairs in the country" but no detailed discussion of Brexit is expected.
After entering Downing Street in July, the PM was quick to hold meetings with key players from the other 27 countries but the European Council summit will be the first time she sits round the table in Brussels with leaders of other member states.
No 10 has indicated MPs will have a vote on the final Brexit deal secured by the Government if the agreement results in a new treaty with the European Union.