09/03/2015 11:40 GMT | Updated 08/05/2015 06:59 BST

Big Data 2.0

Silicon Valley has always had a reputation for moving a little faster than the rest of the world, so much so that I've often referred to it as moving in dog years.

Today's pioneers are far more likely to over-egg the potential of their technology, making grandiose claims that are used to beat the industry when those promises show signs of falling flat: it happened with the web, and television, radio, motion pictures and the telegraph before it.

Sometimes it's a case of waiting as the rest of the world plays catch up, but sometimes there's a genuine need to slow down. We've certainly seen both sides to this trend in the emerging big data industry.

For the last 10 years, the tech world has been talking up the value of big data, which enables vast amounts of information from a diverse array of sources to be aggregated and queried very quickly. It is helping banks massively increase the accuracy of their fraud-detection measures and pharmaceutical companies develop life-saving new drugs, all at a speed that would have previously been impossible.

But while the focus has largely been on big data's promise, less attention has been paid to how companies should go about putting it into practice - because although the majority appreciates its potential, it's a minority that fully understands how to roll out the technology.

I firmly believe that data is our most precious commodity and holds the key to understanding the world around us, but the truth is many are still trying to work out how big data might work for them. And as the big data movement approaches its tenth birthday, there are two clear lessons that have become apparent to everyone working in the field.

First, an appreciation that big data is not some miracle cure that automatically makes an organisation function more efficiently. It requires implementing long-term strategies that have a far-reaching impact on the ways an enterprise approaches operations, innovation and competition. In short, it necessitates making a shift that allows data intelligence to be at the heart of the organisation.

Big data strategies rely on leadership teams that set clear goals, define what success looks like and ask the right questions. And the most effective data solutions identify requirements first before leveraging infrastructure, data sources and analytics to support the opportunity.

There can be no denying that progress is being made in this regard and we're certainly seeing a huge shift at my company, WANdisco - we're now helping three of the world's largest banks implement their big data strategies.

The second issue is a need for greater clarity. In the brief time since a team at Yahoo! developed Apache Hadoop - the underlying platform for big data storage and analytics - the big data market has well and truly taken off. But alongside this lightening growth has come considerable fragmentation, leading to confusion among users.

With an ever-increasing number of suppliers to choose from, companies looking to harness big data are increasingly finding themselves getting stuck: they must either commit to a particular provider, limit their use of big data, or deploy a complex and expensive infrastructure that involves multiple vendors. All this ultimately results in is frustration for customers unable to achieve the results they desire.

This is something that resonates strongly at WANdisco and which we believe demands industry-wide action. It is for this reason we have accepted an invitation to become a founding member - and sole UK representative - of the Open Data Platform Initiative (ODP), a shared effort to promote and advance the state of big data technologies.

ODP will coordinate the efforts of the world's greatest computer engineers from the likes of IBM, EMC, Hortonworks, General Electric and Pivotal, to ensure the benefits of big data reach a broader range of customers than ever before.

Our mission is to accelerate the big data movement by providing a well-defined series of industry standards, building an ecosystem where users have true flexibility and vendors are committed to collaboration. In practice, this will bring an end to what the sector refers to as vendor lock-in - where a business is forced into using only one provider at an early stage and unable to add new features.

This is our attempt to address customer frustrations and we are determined to make it easier for our clients to identify a data strategy that works for them.

It draws a line in the sand for the quickly maturing big data market, recognising the calls for greater flexibility that we hear on a daily basis.

We are at the start of an incredibly exciting new phase in big data's young life. I for one can't wait to see what the next 10 years have in store.