The Big Data Discovery Puzzle

By Kevin Long, Business Development Director, Teradata UK

No matter what the industry, forming the intelligent discovery environment required to generate competitive advantage from big data can be extremely tough. Although traditional analytics and methodologies are fairly robust, in the rush to generate insight from big data projects, the introduction of new analytics technologies and previously unfamiliar open source data storage tools like Hadoop will inevitably create a learning curve.

Creating such an environment also tends to require a shift in mindset. Traditionally, IT culture has always been very requirements-led, making traditional BI, analytics and data warehousing a tough juggling act. Indeed, simply retrieving the data and working out what to do with it required a high degree of flexibility.

Big data discovery projects take this tough equation even further – away from the IT department’s comfort zone. No longer is there a pre-defined ‘need.’ Rather, the challenge now lies in identifying ‘the question.’ And there are many more new and in-depth questions being asked than ever before.

Despite the challenges, the hype around big data has helped to secure buy-in from enterprises as well as prompt IT workers to begin their own experiments that are outside the usual rules of IT. Often, because these technologies are seen as being time-consuming or disruptive, they tend not to get immediately passed to the BI department, who in turn cannot inform IT of existing challenges in the traditional environment that could be solved using the new capabilities.

Yet, even if the discovery capabilities are kept separate, from an execution perspective, it’s still important to find ways to join them together. Where it exists, the most logical approach is for the BI competency centre to drive the big data exploration and execution process.

In industries where individuals and departments work to produce their own analysis of data sets, it can also be harder to ‘stitch together’ pools of information and explore trends or changes over time. This lack of collaboration also means it takes much longer to reach a common solution and understanding of the potential opportunities. Crucially, the process of pooling together data sets can help to make it possible to realise this value.

Some of the best opportunities are extensions of existing opportunities that were previously prevented by cost and processing limitations. In retail, for instance, basket analysis is a well-recognised tool for cross-promotions and marketing. Yet, very little is known about external events ‘outside’ of the basket.

For example, there are many cases when a customer will return to a site to get an item they did not purchase originally, such as matching shoes to accompany a dress or handbag, or an HDMI cable needed for an electrical item. Using sequential affinity analysis, the retailer can capitalise on this increased intelligence to send specific and timely email marketing that’s more likely to drive traffic and increase revenue.

A successful big data discovery environment can also enable retailers to understand customer behaviour better because it allows them to look for changes in an individual’s basket over a period of years, rather than weeks or months. This then allows the retailer to assess the impact, for instance, that becoming a family can have on the purchasing patterns of a previously single customer.

The scope for discovery in big data projects presents countless new opportunities for all sorts of industries. Crucially, if successful, it should provide the means not only to identify previously unrecognised insight but also to process this information to improve quality and efficiency, as well as drive sales.

Kevin Long is Business Development Director at Teradata, the world’s leading analytic data solutions company. He possesses an unusual combination of business, financial and IT skills, which he puts to good use advising how businesses can ‘do more with their data,’ through actionable insight that drives competitive advantage: www.teradata.com