11/11/2015 09:34 GMT | Updated 10/11/2016 05:12 GMT

A Data Revolution - Thwarted?

You might think that, with money tight, Whitehall would be interested in anything that helps the government do more with the resources they already have. That Ministers would jump at initiatives which will let them spend their money a bit better, on things that are tested and proven to work.

You might even think that, with the government making such a noise about the need for open data, they would be itching to get people to use whatever data they have.

Sadly, for the most part you would be wrong.

Crucial information, made up of material about all of us, lies unused in the bowels of government departments and on computer hard drives. It tells us what happens to people across time: how a single individual gains qualifications, moves into employment, what helps them thrive (or not) along the way. That sort of data is gold dust.

One department, the Ministry of Justice, gets this. Back in 2013, the MoJ started some work with NPC, the charity think tank I head. They had the same questions as us: is the work done by charities to rehabilitate prisoners effective or not? There are thousands of these charities but, for all their good intentions, how successful were they in helping prisoners back onto the straight and narrow?

The MoJ swiftly got to work. Led by some inspiring officials, they brought together data on offenders from a mish-mash of public sources, to create a Justice Data Lab (JDL).

For the first time a charity--or indeed anyone else--could present the JDL with the identities of the people they worked with. With strict protections (no one except the charity and the officials would know the names involved), the charity would then have data on how many have re-offended, how often, and the severity of the offences if they did.

At a stroke, this is more information than most charities will ever have had before. No amount of surveys and follow-up queries by the charity would produce it. Even better, the JDL could create a control group of prisoners with similar characteristics who didn't get any help from the charity, to give a good picture of whether the intervention made a difference.

Immediately, charities and policy makers would start to understand better what works and what does not. It's a win-win, and it costs very little.

You might expect that this sort of idea would have spread like wildfire. But not at all. The Justice Data Lab is still the only place where civil society can get consistent access to government data.

There have been twitches of interest elsewhere, but no more than that.

To his credit, Iain Duncan Smith broke through an impasse in his department about a similar initiative, by announcing his plans to develop a data lab just before the election. But the department has since chosen to hide behind a myriad of genuine but hardly clinching problems, rather than working out how to solve them. Very little progress has been made.

In health, where we so badly need to find ways to save money and improve care standards, there has been some progress. A version of the data lab is being used to assess the new pilots (known as Vanguards) set up in the wake of Simon Stevens' Five Years Forward View plan for the NHS. But this still feels a long way from a health data lab available to all, through which we can identify the most cost-effective ways of keeping people out of hospitals if they neither need nor want to be there.

Of course, data throws up problems, to do with privacy, data-sharing, user-consent and how we end up interpreting the results. But as an ex-government hand myself I'm pretty sure that, with the will, officials and ministers would find a way.

I wrote to the Chancellor this week with precisely this message. The political will of the treasury has a useful way of unpicking problems, and smarter use of data to save money and prove what works in improving lives is badly needed.

In the meantime, the charity sector itself cannot escape its share of blame for this state of affairs.

If we think again about criminal justice, there is now a free tool to help charities assess their impact. Yet surprisingly few--not much over 40--have used it. What are they waiting for?

We know that some charities are scared of discovering something they don't want to know: that what they do might not, in fact, reduce re-offending. That's a risk, and while the money is flowing in and the volunteers are happy, why rock the boat?

But this just ain't good enough. Every charity board in that area should be demanding of its CEO, are we using the JDL; and if not, give me a bloody good reason why not.

Data has the power to transform our ability to solve social problems. The government has stacks of it, collected from its citizens. Now is surely the time to use it.