It's recognised that big data not only exists but that it has great potential to disrupt well-established vertical markets. IBM commented on information increasing twofold while Cisco predicted that by 2015, there will be 15 billion connected devices in circulation. While, until recently, the big data phenomenon was a hype generated by the technology industry and media, it has been rapidly unfolding and growing behind the scene.
How do we know this? Because all industries and sectors are expected to have real-time analysis of what's happening. In the same way that customer service has evolved to be 24/7, real-time data analysis for better opportunity and anomaly spotting is something that the majority of us take for granted.
Let's take a couple of examples. In retail for example, have we not come to expect Amazon and other online retailers to recommend our next purchase based on prior buying habits? It's quite telling that Amazon claims 30% of its sales are generated as a result of its recommendation engine: "customers who bought this book also purchased..."
While most consumers now take it for granted that they receive personalised and tailored offers in the retail, utility and telecoms industries, it's not as welcome in other vertical sectors, for example public sector or government-based industries.
Those industries and vertical sectors that have easy and available access to customer or industry data to compare and contrast should be taking advantage of this. At a time when cost comparison and running as leanly and efficiently as possible is crucial, all companies need to be looking for opportunities where they can also benefit and find growth. And many are.
As mentioned, the retail sector in particular is doing well out of analysing customer behaviour and patterns and being able to offer consumers coupons almost in real-time. Two great examples are UK-based retailers, sandwich shop EAT and Sainsbury's. Using business intelligence technology to pull together data on ingredient purchasing, weather, store visits and staffing, EAT has been able to purchase seasonally and staff according to demand. The other example is Sainsbury's, which is using big data to help it set prices, nearly in real-time, and shift inventory by giving loyal shoppers customised coupons.
There are definite similarities to be drawn between the retail space and the telecoms and mobile industries. What the latter has an advantage on, however, is the consumer and user data. Thanks to (mostly) long-standing customer relationships, the billing history can throw up a huge amount of insight on the individual mobile user and their habits. In fact, this information is so valuable that European mobile giant Telefónica last year was set to launch a global big data business unit aimed at selling information on customers using its mobile services. Some of the immediate benefits on offer would have appealed to the likes of local councils who want to measure how many people visit the high street after the introduction of free car parking, farmers' markets or late night shopping for instance.
However, it throws up major privacy concerns for the customers and any major business admitting to selling off the data it analyses. All too often, many of the businesses are not willing to risk their customer relationships in return for better knowledge and partnerships with other organisations that could further improve the service or customer experience provided.
Another vertical sector where big data has huge potential but regulatory issues often come up against the potential is in the financial and banking industries. While over 2,500 financial services institutions globally have signed up to use our software, utilising it to continue to manage risk, meet regulatory compliance, spot growth opportunities and increase margins, with users spanning many departments - from Risk Management to Client Servicing - there is still far more that can be done if silos were overcome and business users given access to the data for analysis. Facilitating better, faster decisions and arming employees with the correct tools, the route to success with big data analysis could be made as simple as possible, if weren't for complex regulations and an unwillingness to share what is perceived as competitive insight
Yet, it was information asymmetries that led to many utilities and retailers entering the financial services marketplace in the 1990s. They rightly argued that they had more potential information about the customers of the banks than banks had about their own customers. Just how successful these diversification attempts have been remains to be seen. Recently both Vodafone and Apple claimed they had considered or were considering opening banking arms but these have yet to have occurred.
What the big data potential does demonstrate, however, is an opportunity for open innovation. By empowering all the users within a business, and the possibility for change, innovative transformation should theoretically become feasible. Not only do organisations in all sectors have to remain agile and able to adapt quickly to stay ahead of the competition in their specific vertical sector, but also as companies' traditional roles evolve. This comes back to the diversification challenge and opportunity - and several failed attempts.
On the other hand, it is these information asymmetries that are enabling technology companies such as Google and Microsoft to do just that. Diversify by moving into the healthcare market, allowing customers to track their health and record their treatments. The same can be said for the Internet of Things and in creating a smarter world reliant on the connections around us but also the analysis of these, How else do we move from electronics companies creating smart fridge that speak directly to your preferred retailer to order you milk and yoghurt when you've run out?
Before we get ahead of ourselves and move into the world of the future, let's consider some other vertical sectors around the world. Namely the utility sector, which is embracing big data to analyse customer sentiment online and in social networks. And here is an example of where one sector is more advanced in certain regions than others. Utility companies are using data through social media analysis more effectively in the US and Europe than in the UK. This again shows that there is still more potential to be harnessed, even if only in certain regions.
As we touched on earlier, the private sector has arguably greater licence to analyse user patterns to better tailor and personalise services, because customers opt in to a brand. That said, governments and the public sectors around the world are not letting the big data opportunity pass them by. Police departments are using computerised mapping and analysis of variables like historical arrest patterns, paydays, sporting events, rainfall and holidays to try to predict crime 'hot spots' and deploy officers there in advance with big data analysis. From Stockholm to the UK and across the US, police forces are being very effective in managing their resources for better deployment.
Opening up data is key to this, as is being transparent in how data is being stored and analysed. The UK public sector is driving this with 'Open Data' initiatives, giving interested parties access to data and, with the right tools, there's no reason why users cannot draw their own analysis and reasonings, making for a more engaged society.
Which bring us back to what big data is and where its responsibility sits. One thing big data giant EMC advises against in any case is treating any initiative to address big data as an IT issue. While the IT department and so-called data scientists have an important role to play in enabling access to data access and then drawing analysis from multiple rows of data, it is down to the business users to spot anomalies and opportunities from the data available to them. The more we can refer to big data as information that is available and the business insights harnessed from the analysis the opportunity for different vertical sectors all across the globe, the better placed we will be to take advantage of what big data ultimately enables.
The best big data is the data generated as a by-product of operational, customer and supplier processes. The data that people naturally share, and are willing to, in return for a better experience or end product. And the best big data is when it becomes information that is readily analysed by business users for useful insights.