A Conservative defence minister has said the "disgraceful" campaign by Joanna Lumley on behalf of the Gurkhas has led to some of his Aldershot constituents leaving the town.
Gerald Howarth, the minister for international security strategy, said the number of Nepalese men who had served in the British Army deciding to move the the UK was putting too big a strain on local services.
In 2009 Lumley, with the backing of David Cameron and Nick Clegg, helped convince the then Labour government to allow Gurkhas who retired before 1997 the right to settle in the UK.
But speaking on BBC Radio Surrey on Monday, Howarth said the decision had been a mistake and was adversely affecting his constituency, which is the home of the British Army.
The Aldershot MP said: "I was walking around in Aldershot on Saturday and everywhere I went there were Nepalese just basically sitting out in the open, sitting out on the park benches.
"You may find this bizarre, but some of my constituents say 'if I go into town, I haven't got a park bench to sit on because they are all taken by the Nepalese'."
Howarth said the problem was not with Gurkhas who had just left the army as they tended to be "young, fit and very fluent in English" as well as "very qualified" and able to find work.
Rather it came from the number of Nepalese coming to the UK who had left the British Army as long ago as the 1970s who were now elderly, in poor health and who did not speak English very well.
He said: "The problem arises because in 2009 the government decided, in the face of the pretty disgraceful campaign by Joanna Lumley, I don't entirely blame her, but it was an entirely emotional campaign, which resulted in all those who had left the army before 1997 given the entitlement to come to the UK."
He added: "They don't understand our customs."
Howarth said his constituents were "absolutely fed up" and some had told him they would leave Aldershot if the situation did not change. "If they say something they get accused of being racist, it's their town," he said.
"It's not fair on them and it's not fair on the elderly Gurkhas," he added. "The people of Aldershot and Farnbourgh require a solution."
Peter Carroll, founder of the Gurkha Justice Campaign, said the comments showed a "complete contempt" for the veterans.
"His comments about 'park benches' show an attitude of complete contempt for the very brave men and their families who now live in his area," the Press Association quoted him as saying.
"He is talking about soldiers, some of whom will be veterans of the Falklands and other conflicts, as if they were a nuisance.
Kevan Jones MP, Labour’s shadow defence minister, said Howarth had shown a "worrying lack of respect" to the former soldiers.
“We recognise the pressures on the community in Aldershot but Gerald Howarth and the Conservatives supported Gurkhas in Opposition and should do so in government," he said.
“This is a government making mass redundancies and who desperately need to show some compassion towards service veterans.
He added: “Mr Howarth should clarify his comments and apologise for any offence caused.”