The thorny issue of whether foreigners who commit crimes in the UK should be deported immediately surfaced in the House of Commons on Monday afternoon. The debate was triggered by revelations in a leaked report from the UK Border Agency, which showed many foreign criminals had remained in the UK despite attempts to deport them, and some had been arrested subsequently for serious offences including rape and murder.

In an urgent question tabled by Labour's shadow immigration minister, Chris Bryant, the government said it was doing all it could to remove foreign criminals from British soil, but this was being blocked by judges on human rights grounds in many cases.

Immigration minister Damian Green told the Commons that there were nearly 4,000 foreign criminals in Britain awaiting deportation - a reduction of 1,000 on recent figures. Some of those remained in jail, others had been released after serving their sentences.

"We are doing everything in our power to increase the number and speed of the removals,' Green told the Commons, saying the coalition government had begun the deportation process for foreign criminals 18 months before they were due to be released.

However the subsequent debate triggered several questions as to whether foreign criminals should serve their full prison sentences at all, with MPs Labour and Tory alike suggesting that they should just be deported as soon as possible, regardless of how far into their sentences the convicted foreigners were.

MPs are mindful of the costs involved with incarcerating prisoners, which is never less than £30,000 a year, and can often exceed £100,000 a year.

Damian Green said many home secretaries over successive governments had wrestled with the notion of sending foreign criminals home before they had completed their sentences, but told the Commons that both Labour and Tory governments had taken the view that failing to imprison foreign criminals for a full stretch would lend the impression they could offend, safe in the knowledge that they'd go home sooner and be free.

Later in the debate Tory MP Philip Hollobone asked the minister to confirm newspaper reports yesterday of the most awkward countries to deal with on deportations.

Green paid tribute to Jamaica and Nigeria - two countries from which many foreign criminals in Britain originate, saying both countries had become more co-operative in recent years. "They are not the most awkward," said Green, declining to say which countries were.