So, it's now been said a million times in a million ways: big data can predict the future. My problem and the problem of many, I suspect, is that one can't help asking - so what? Especially if the future is going to be unpleasant, whether in be in the economy, in geo-politics, in human health or even just in the weather, what is the point in knowing?
Of course, wanting to know is human. The ability to accurately predict what's coming has been a fascination of humankind for as long as history can record. From the biblical Book of Revelations to Nostradamus' Prophecies - colourful tales describing the future have been big hits. This interest grew to something of a frenzy in the 20th Century - which saw a huge output of science fiction and dystopian futuristic novels. These later books were more realistic, and many predictions have already come true. Think Jule's Verne's Twenty Thousand Leagues Under the Sea's prediction of submarines or George Orwell's in places scarily accurate Nineteen Eighty-Four, in which everyone is constantly under the watch of Big Brother.
But my personal favourite futurologist, science fiction author, Mensa member and professor of biochemistry went one step further. Isaac Asimov's brilliant work, the Foundation series, engaged with the subject of the future in a different way. He showed that the real trick is not to treat the future as a fait accompli.
In the Foundation series a new form of mathematics known as Psychohistory is developed to predict the future of humanity, on a galactic scale.
Psychohistory collects large amounts of information from diverse areas of study such as history and sociology are combined in a mathematical formula to predict the future of the galaxy. In the series, the applied formula outlines two possible futures. The first is a dark age of 30,000 years between the fall of one empire and the start of the next. The second predicts a dark age of just 1,000 years.
With this information, the leading scientists of the age create a plan to ensure that the 30,000 years of darkness scenario is avoided. The eponymous "Foundation" is formed to guide humanity towards the preferable outcome, and to start a new galactic empire.
Back here on Planet Earth, Big Data and its potential to use historic data sets to predict the future is shaping up to be the story of the century. Organisations from rating agencies and investment funds to governments and meteorological centres to space agencies, health providers and employment agencies are gearing up to get a better picture of what's happening and to foresee the future based on analysing past data.
But is this really the best use of Big Data? Are predictions really all that useful or is it the data and assumptions behind it that we really need?
At present Big Data is geared towards to a 'give me the results' outlook. With very little thought given to presenting the processes or alternative theories behind the predictions.
But it is what lays behind the headline predictions that will actually have an impact on what we should do today. If a political scientist, for example, used big data to predict that within three years of becoming President, Donald Trump would crash the United State's economy - the only way to prove his or her prediction was correct, would be for the crash to happen. Or not.
However access to the assumptions, data and reasoning that led to this prediction could help people decide how to vote today.
The reluctance of organisations to share what backs up the predictions they are making with Big Data is understandable. The huge amount of resources given to Big Data predictions results, effectively, in proprietary information. Why would a company want to share this with its competitors?
But why is big data derived information treated as so deeply precious? After all, banks share their interest rate structure. Public availability helps the entire industry. Other sectors, likewise, share information as a matter of course.
And it's through sharing the assumptions made and even the theories that are considered unlikely, and the end results themselves, that Big Data predictions can have a real-world-use case.
In Foundation the information used was galactic. On Earth we may not be dealing with anything as large as the future of the galaxy but Big Data, used usefully, can help shape our present and have a real impact on shaping the future.
But only if we move away from hoarding data for ourselves and let the world use it. Simply put, if we can predict the future, we can help change it.
Whether humanity learns to share in time, is anyone's guess.
As Asimov himself said:
"The saddest aspect of life right now is that science gathers knowledge faster than society gathers wisdom."