Both Government and the public sector are in the enviable position of having lots of data about UK citizens. However, the current culture, systems and processes make it difficult to unlock the insight held within that data to help improve services and drive efficiency. Thankfully, this is now likely to change given the announcement earlier this month that the Government is vigorously reviewing legal barriers to data sharing.
Given there is also affordable technology available to take advantage of big data, there are few excuses for public sector organisations to underuse this valuable asset. Yet draconian measures taken to isolate data from other parts of the public sector still apply, despite the benefits that would arise from integrating data across departments to give a joined-up view of people, operations and services.
Imagine every time a citizen had contact with Government that one single view of that citizen was available, rather than the citizen having to give the same details every time. Imagine that when people came off benefits it was possible to examine their wider profile; how long they may have been in housing, how many children they have, the children's ages, types of benefits and what factor was the trigger for them to come off benefits. This insight could help prevent people in similar circumstances going onto benefits in the first place.
Big data analytics can be used to understand what data is important, and transform it into insight for better decision-making, increased efficiency, faster innovation and improved services.
Of course, enterprise analytics is not just a question of data volume. It's about variety; comprising a mixture of traditional, structured data, as well as unstructured files and forms such as social media and emails. It's also about velocity; the unprecedented pace at which data may need analysing.
The velocity of digital content means government must analyse data and make decisions using technology. However, the data may not need to be stored - it could be data that needs to be accessed temporarily to identify an emerging threat, for example. The data might only be needed for long enough to deal with the threat - e.g. passenger medical and travel data to prevent the spread of a deadly contagious disease. Global pandemics did not exist before air travel which has changed inter-connectivity and increased the risk of killer diseases being transmitted rapidly across continents. Digital analysis will spot potential threats before humans do, which could ultimately save lives.
While the concept of big data is nothing new, there are a number of outdated perceptions that prevent Government from making the leap from lots of data to the big data dream.
Myth #1: Big data management and analytics are expensive
In the past, storing, managing and analysing big data could be prohibitively expensive for cash-strapped governments, but technology has developed. Hadoop, the open-source software framework which allows for massive data storage and faster processing, has reduced storage costs. It can store huge amounts of data by breaking the data into blocks and storing it on clusters of lower-cost commodity hardware. By enabling analysis across data for the entire population, for example, rather than a sample selected in some way, more accurate insights follow as they are less likely to be skewed by inappropriate sampling techniques. The best statistician in the world could not achieve the same results.
Myth #2: Big data analytics takes time and specialist skills
Visual analytics, drag-and-drop dashboards, and apps for mobile devices mean anyone can explore big data to reveal previously hidden patterns. Users with no analytical training have access to the power of predictive analytics to generate answers immediately.
Myth #3: Public data cannot be shared
This is 'data fear' rather than a legal barrier. There are various options for how greater data sharing could be achieved. One might be to introduce a system where citizens are asked what data they would be prepared to share. Gradual sharing of data, where relevant, could be achieved by first assessing the usefulness of data to different departments, so as to assess the merits of joining up data sets versus risks to civil liberties. Even if data itself is not shared, it may be possible for one department to carry out analysis of its data and share insights from that analysis with other departments. There could also be sharing of models - there would again be no need to share data, as it could be a case of simply running the same model on another department's data.
Myth #4: Public sector workers are too busy to collect data and data collection is expensive
Government must review processes to ensure collection is automated where possible and made available in a meaningful way that is useable by frontline staff. Yet, by fully embracing the use of existing administrative data, it could save time and money, remove the need for sampling, and deliver more up-to-date insight, faster.
There may be a need to collect other information, beyond what has already been provided by citizens. A simple, centralised system for understanding which departments already hold that information could reduce the burden on collecting data and avoid duplicated effort.
Myth #5: Public sector has a robust decision-making culture
In the digital age, people understand that the data and technology are available to make evidence-based decisions that stand up to scrutiny (from both the public and press). As awareness of the power of big data grows, the public sector needs to develop a culture of transparency, providing data-led evidence around spending, services and policy.
Abolishing these myths is the key the Government and the public sector needs to unlock the valuable data it has locked away.