Navigating The Big Data Ups And Downs With Exploration And Discovery

Some people seem conflicted about big data at the moment.  On the one hand, excitement about the benefits of big data is at an all-time high. In the Analytics 2013 survey of 600 analytic professionals showed that 35% of companies were expanding their analytic investments because of big data.  Big data represents a world of opportunities to take advantage of our ability to measure just about anything and a wealth of new data sources including machine generated data and unstructured data sources.  On the other hand though, I’m starting to hear more skepticism.  I was recently asked, “Why is everyone so grumpy about big data?”  That question led to a discussion and white paper about the challenges some have had in generating a positive outcome or ROI from their big data projects.

I’m a believer in the potential of big data.  At Lavastorm Analytics our data analytics platform has allowed me to see the benefits first hand, such as an organization that pulled together 60 different data sources into a single application to identify and eliminate fraudulent activity and another financial services organization that analyzed terabytes of data daily to improve the effectiveness of their trade execution.  I expect its ability to improve customer relationships, change business models, and improve overall decision making will be borne out in large scale over the next 5 years.

But while big data has high potential, it would be a mistake to think that simply add a new data source to your mix would immediately reveal the secrets to the world’s longest lived mysteries.  I believe any skepticism at this point has to be routed in poor processes applied to big data, not the big data itself.  It isn’t obvious in every industry which data is the most valuable.  You “could” analyze thousands of data relationships, but not all will bear fruit and if your peers haven’t already done something you can just emulate, then we have to recognize that the first task is identifying which data and relationships are the most valuable.  That cries out for an exploration and discovery approach – one that allows you to initially cast a wide net at minimal cost so that you can simply identify a handful of the most likely candidates that are worthy of greater investment.  Selecting a low cost, low risk infrastructure or toolset to enable that exploration and discovery of those targets is a major key to success.