#bdw12 What was it all about?

“It’s all about Big Data” we promised, and for one week in April, that was certainly the case. From almost any angle of computing – technical, managerial, theoretical, artistic – data has never enjoyed so much attention. With attention comes caution – “How do we manage so much stuff?” – and, thankfully, Big Data Week kept one eye on these pressing, contemporary issues as well as the other firmly focussed on the future.

The cornerstone of the week was the Data Science Hackathon: the World Tour of hackdays, if you will. Taking place across 48 hours in three continents, a phenomenal 114 project submissions were made. Given that this was a Hack with big data, the 48 hours certainly appear to have presented a physical and mental challenge; one hacker exercised frustration at the inability to create a 5.2GB vector. However, given the shadow that Moore put over the speed of technological change, it won’t be long before 5.2GB feels like a floppy disk to consumers. This means that the big data hacks of the present will be tremendously important in the future: shaping and contextualising the potential of what we can do with silicon as price scales at an inverse level to capacity.

Presenters, delegates, and commentators from across the week shared a wealth of insight into what “big data” means to them and to the way in which we conduct business. Data scientists clearly find big data something of a welcome relief; gone are the days of sampling because it is impossible to use an entire dataset. However, working with entire datasets also means working across departments and silos – something that big business   isn’t necessarily used to. This suggests that regrouping to cope with big data isn’t just about dealing with the technical problems, but dealing with structural and operational issues before they become problems that the best technology in the world cannot fix.

Data visualisation has gained attention outside of the realms of data science, and the value of dataviz was not forgotten. While the debate continues as to dataviz being a tale of “volume over value”, there’s no question that what dataviz presents is the ability to tell deep, meaningful, compelling stories to C-suite executives. It’s the perfect combination of science and art.

With opportunities come concerns. As is often the case with digital technology, the unchartered ventures of today become subject to the draconian measures of tomorrow. Observations were made as to what the Government’s view of big data should be, with some suggesting that all big data will be regulated; for consumers, this could mean controls on the level of data that is held by Facebook or Google. While this may feel like a good thing, it’s inevitable that there will be commercial pressure from the new global players – Zuckerberg, Brin/Page, Cook – to argue that, for them, big data offers commercial opportunities (such as contextual display advertising, or brand insights) that tightly-controlled data management many not. Perhaps, as some discussions suggested during the week, we should see data in a different way – as a physical entity, rather than a fluid, massless, collection of atoms (or something like atoms). One of the key reasons as to why a different view of data was suggested was to identify a need to avoid the sheer volume of data-based lawsuits that currently exist; good news for lawyers, bad news for those at the sharp edge of what this all means.

Specific sectors were also addressed; while researchers and scientists are embracing the potential that big data offers, the value is also very much there for HR, finance, and in consumer-centric operations such as billing, below-the-line advertising, and planning. The most agile businesses will find big data to be of value in deriving new, more precise, outcomes in both strategy and delivery; as long as this value is manifested to customers, then it’s a great investment all round.

The concepts of big data are increasingly commonplace, to the extent where they will become absorbed into the fabric of business in coming years. There are new challenges, but they are not impossible to overcome. We’ll be seeing the sharp thinking that has come from this extraordinary global event transferred into the day jobs of many to come. There’s no doubt that big data will have a big impact.