David Cameron is seeking to change the European Court of Human Rights (ECHR) so that it no longer serves as an automatic last resort for criminals, including foreigners awaiting deportation. But the proposal has been attacked by Labour who believe the prime minister may be paving the way to "walk away" from the court.

Cameron will use Britain's presidency of the Council of Europe to attempt to make changes to the court. However Downing Street has admitted that getting agreement on how to speed up the Court's operation will be an enormous challenge.

The ECHR's remit covers 800 million people in 47 countries and now has a backlog of more than 150,00 cases.

In a speech in Strasbourg on Wednesday the prime minister will say the court should be free to deal with the "most serious violations" of human rights and should not be "swamped" with an endless backlog.

"The court should ensure that the right to individual petition counts; it should not act as a small claims court. And the Court should hold us all to account; it should not undermine its own reputation by going over national decisions where it does not need to," he will say.

In a clear sign that Britain is no longer prepared to tolerate the European Court being an automatic backstop for those convicted in the UK, Whitehall sources have indicated that a detailed draft of proposed reforms will be ready by the end of February, with a conference of European justice ministers being convened in April.

Britain holds the rotating presidency of the Council of Europe until May, reflecting the speed at which the PM is seeking to make these changes. Whitehall is expecting a series of "short but intense" negotiations on the reforms, and while there is agreement within the coalition and among many European nations that the court is unfit for purpose, it is far from clear how reforms which can quickly separate genuine human rights abuses from spurious cases can be achieved.

In a sign of frustration in Number 10 with the court's interpretation of its powers under the European Convention on Human Rights, the prime minister will also say: "We are hoping to get consensus on strengthening subsidiarity – the principle that where possible, final decisions should be made nationally."

The current wait for a case to be heard by the court is around two years, leading to many would-be deportees languishing in jail. The system leads to long-running and emotive cases, including that of the Islamist cleric Abu Qatada, whose deportation to Jordan was blocked by the European Court on human rights grounds last week. The case caused Tories to call for a withdrawal from the European Convention on Human Rights, although what's being proposed falls far short of that.

Downing Street sources spoke of a "single agenda, which is reforming the court." There is no apparent move by Britain to amend the European Convention itself, with government sources suggesting that even making the changes outlined will be "very difficult".

There is nothing in the coalition agreement on changing the ECHR, but sources close to Nick Clegg said the deputy prime minister was entirely supportive of Cameron's plan. "Nick has been calling for reform of the court for many years, and as with all speeches this was agreed beforehand," said one source.

Labour have criticised the PM's rhetoric on reforming the ECHR, suggesting it could be the first step towards a unilateral move away from its auspices if the negotiations fail.

Shadow Justice Secretary Sadiq Khan accused the PM of "peddling myths that denigrate the human rights successes of the Court and the Convention. It smacks of throwing ‘red meat’ to the hungry pack of Conservative backbenches so recently emboldened by the Prime Minister’s waltzing away from the European negotiating table."

He went on: “The Labour Party is fiercely proud of the UK’s role in protecting and championing human rights across Europe, and beyond. Other countries look up to the UK, and our moral authority as a member of the club of 47 nations empowers us to pressure those who have weaker human rights records. To simply walk away, as the prime minister has hinted, would constitute a gross neglecting of our duties as a beacon of civility amongst the family of nations.”

The human rights group JUSTICE has also criticised Cameron's comments. Angela Patrick, director of human rights policy said some of the cases the government does not like deal with issues that are "far from small" involving "at their heart a fundamental commitment to an absolute bar on torture".

"Comparing the Strasbourg Court to a small claims court damages our public commitment to the international rule of law," she said.

Conservative MPs reacted with glee at Labour's instant opposition to the proposals, with one telling HuffPost UK they were finding it hard to believe that Ed Miliband was managing to be out of touch with public opinion on another issue.

"Who's advising him?" said one Tory MP, who was entirely supportive of the plans.