Appealing to younger voters, former Conservative minister Liam Fox has said that mass immigration places pressure on housing, while increasing rents and house prices.
Any further “uncontrolled” migration would worsen the plight of prospective homeowners, he said.
Remain campaigners described the belief as “fantasy”.
Fox said on Thursday: “Most new immigrants move into the private rented sector which has grown as the immigrant population has grown.
“Competition for rented accommodation obliges all those in the private rented sector to pay high rents which take a large share of income and makes saving to buy a home even harder.
“These resulting high rents and a shortage of housing make it much more difficult for young people to set up home on their own so they have to spend more time in house shares or with their parents.”
However, property expert Henry Pryor, told The Huffington Post UK that it is too simplistic to say that leaving the EU would make property more affordable.
“If we come out there will be uncertainty and house prices are predicated on the cost of credit,” he said.
“If we come out there is an argument that the cost of credit will rise as inflation will increase more quickly than if we were to remain.
“This referendum is not easy - we can’t be sure of either outcome. We can say markets don’t like uncertainty.”
Meanwhile Remain campaigner and Liberal Democrat leader Tim Farron attacked Mr Fox's argument, saying Brexit would "destroy young people's hopes of getting on the housing ladder".
He continued: "Leaving would mean fewer jobs, higher prices and lower pay, making deposits harder or impossible to build up. And it means higher mortgage payments making first time loans less affordable.
"First time buyers are better off in Europe and leaving would be a leap in the dark leaving young people worse off.”
Housing is a key concern for young people, with research by Bite The Ballot and Hope Not Hate finding 27% of under 25s saying it is their top priority.
During a heated BBC debate, a young woman lamented restrictions to housing which she believed were caused by high immigration.