On 12 June the Ministry of Justice (MoJ) published interim reconviction figures from the reducing reoffending payment by results (PbR) pilots at Peterborough and Doncaster prisons. Unsurprisingly the Ministry, as well as Social Finance who are behind the Peterborough social impact bond (SIB), have highlighted the positive angles of these initial findings, and one cannot deny that they are at least encouraging--though it is early days. However if you've read any of my previous blogs on social impact bonds, social investment and payment by results, you'll be unsurprised to hear that although a fan of these mechanisms I have a few reservations.
Figures for Peterborough show a 6% fall in the frequency of re-conviction, in the context of a 16% rise nationally, which looks impressive. We must be cautious though - this comparison is not of course one against a proper control group, let alone a randomised control trial, so it could be easy to over claim. Participation in the programme has been optional for prisoners and even thought final results are to be measured against the full cohort, the results are bound to be affected by some sample selection bias. Furthermore Peterborough prison was carefully selected for this project, the governor is very engaged and most prisoners are released into the local area (rather than being dispersed around the country)--all factors which will have enhanced this pilot's chances of working well.
There also seem to be some strange things going on in the national population that the MoJ compares the SIB pilot results to. There seems to be a very steep rise in frequency of reconviction events between the two periods September 2008 to March 2010 and September 2010 to March 2012. We see the same pattern with re-offending rates. Both these trends seem surprising especially after a fall in the reconviction rates in the earlier period, and this probably needs further investigation before we take it fully at face value as a 'fair' comparator.
Unsurprisingly less has been made of the results from the Doncaster pilot where the payment by results system has not been delivering the same apparent level of success. Here they adopted a binary measure of success, and achieved a 0.5% fall in re-conviction levels, compared with a national average fall of 0.3%. Of course we don't know if these are good or bad figures given the lack of control group but these results do not give so much cause for optimism.
It is of course great news that this data is being made available and that so far it shows things are going in the right direction- though how well is debatable. However you choose to construe these initial results the most important thing is to keep this data coming so we all have a chance to try to work out what is working and what is not. We need to take our time with this process and not rush ahead before we really understand this.
Meanwhile, since the real test of impact is with a matched control group, charities interested in this can and should use the new Justice Data Lab (JDL), which MoJ launched and NPC helped design and advocate for. The JDL helps produce good analysis without the need for an independent assessor and lots of data processing--JDL does this free and in the space of a few weeks. .