Saturday, 1 March 2014

The Big Data Revolution: What Happened to Data Quality?

There's a wonderful irony in the world of Big Data Analytics. At a time when interest in Big Data appears to be growing exponentially, it appears that some are forgetting the fundamental challenges of Data Quality. A quick Google Trends analysis highlights the point. The chart below shows two trend lines extracted from Google Trends. The line in blue reflects the popularity of searches for Big Data, while the line in red shows the popularity of searches for Data Quality. It is important to note that the lines show relative popularity, not absolute volumes of search terms. In fact, Google Keywords suggests that in absolute terms searches for Big Data are about 20 times as popular as searches for Data Quality.

This raises an interesting question - what's happened to Data Quality? At a time when organisations are becoming ever more interested in using their data to create performance insights and predictions, interest the Data Quality appears to be declining. Is this because Data Quality is no longer an issue?

I don't think so. On three separate occasions in the last week alone I have been involved in discussions with senior managers from some of the world's leading manufacturing and service businesses. Each time, the issue of Data Quality has come up loud and clear. These firms recognise the potential of Big Data and Analytics, but are realistic enough to know that unless they sort out their data fundamentals - unless the track the right things and make sure the raw data if accessible and of high quality, all of the Big Data Analytics in the world is not going to help them. That's why - in the Cambridge Service Alliance - one of our projects this year is focusing on creating a data diagnostic - a methodology that can be used to check whether the data you have access to is appropriate and can be better used to optimise the delivery of your services and solutions. We're in the process of testing this data diagnostic at the moment and would love to hear from you if you'd be interested in being one of the pilot test sites.