More than 5,000 foreign criminals who should have been deported remain in the UK, including almost 4,000 who are free on the streets, figures showed.

The UK Border Agency (UKBA) must reduce the number of its decisions which are overturned, mainly because of human rights claims, and end the cycle of appeals which bump up the cost to the taxpayer but still fail to see the offender deported, inspectors said.

But the warning comes after Home Secretary Theresa May said she wants to change the rules which prevent foreign inmates from being deported once they have served their prison sentence because it would breach their human rights.

The real problem lay with the way the British courts interpreted the law, she said, adding that the right to a family life was not "absolute" and could not be allowed to "drive a coach and horses through our immigration system".

John Vine, the independent chief inspector of the UKBA, said the number of foreign criminals who are not deported or cannot be deported at the end of their sentence, was increasing. "More must be done to actively manage these cases - they represent a growing cost to the taxpayer and cannot be ignored," he said.

Inspectors found a total of 3,775 former foreign national prisoners who should have been deported had been released from custody and were living in the community.

More than 1,600 others remained in detention, having completed their prison sentence. A further 12 are missing after either being released directly from court or referred incorrectly, he said.

The report found "significant disparity" between the UKBA's and the courts' interpretation of whether a foreign prisoner should be able to remain in the UK on human rights grounds.

A total of 425 cases had been overturned, "the overwhelming majority on human rights grounds", compared with just 151 cases in which the offenders had been granted permission to stay in the first instance. In all, a third of the appeals against deportation lodged by foreign prisoners in the 12 months to February were successful, the figures showed.

Foreign prisoners were also being held for longer, up to 190 days in January from 143 days in February 2010, and one in four who were held after the end of their jail term had been detained for longer than a year.