Tuesday, 29 March 2016

Difficult / Problematic Data and the 3 Vs



As previously mentioned the name of this blog, difficult data, is down to me misremembering a quote when I set it up.  The quote I was going for was from Doug Laney in his 2001 paper on problematic data where he set out the 3 Vs of data that are difficult to handle in traditional database solutions.  Laney's paper is archived at his blog and there is a link at the foot of this post, it's well worth a read, but essentially it set out the 3Vs that are considered standard frameworks for Big Data (this was before Big Data was a thing).
                                                                                                                           
3Vs
Volume is the most obvious of the three and the one that people seem to get hung up on when they are talking about Big Data (Well big is in the name).  The idea here is that with huge amounts of data the traditional database architecture has problems coping and by traditional we mean relational.  The problem is that this is probably the least justifiable of the 3 Vs in most Big Data applications.  The size of data needed before an enterprise Relational Database Management System (RDBMS) starts to struggle is way beyond the vast majority of applications, until the data collection is headed into the Petabyte territory there isn't a huge justification for moving from a standard database arrangement however the other 2 Vs are often more common and useful.

Velocity, the second of the Vs looks at the idea of data coming in faster than we would normally deal with, particularly if this data has limited long term value.  As we start to look at Internet of Things (IoT) applications where we could potentially have hundreds of sensors collecting real-time data many times a second this becomes apparent.  Although the total volume being collected could easily be well within a normal RDBMS capability having all that data available quickly and from concurrent sources is quite a challenge, the problems with sharding RDBMSs are well documented.

The third V is variety.  Here we are concerned with rich datasets that could potentially contain images, sound, video, drawings, tweets, documents etc. These types of data are very difficult to represent within a relational framework where every database entry is an atomic intersection of an attribute and a tuple, even more sophisticated object relations simply allow us to store the item as an object but still not break it down any further within the database itself.

4 and 5 Vs and why it misses the point.
Since Laney’s paper we’ve seen the growth of Big Data which tries to address some of the failings of traditional RDBMSs and small scale analytics (and replaces them with a few of its own).  Laney’s descriptions have been largely taken as the definition of suitable areas for implementing Big Data solutions, any time you have a problem with one or more of the 3 Vs then it’s time to consider a Big Data implementation.  Of course once a thing gets going we can’t just leave it at that now can we ?  You are only using 3 Vs still ? Psssht!

Marketing departments at certain companies (I’m looking at you in particular IBM) have expanded the original 3V concept further into 4 and 5 Vs.  Now here the additional Vs are Veracity (how accurate the data is) and Value (can you actually make money / value from the analysis).  On the one hand there is no doubt that in order for a Big Data implementation to be worth doing the additional Vs are necessary.  No one would argue that a successful implementation needs to have data that is accurate and produces output that reflects reality, and of course any analysis worth producing should of sufficient quality to act upon, there’s no point just analysing for analysis sake (even me, a mere academic, understands that).  Where I take issue with IBM et al here is that they appear to be casting this as a uniquely Big Data thing.  Suddenly we don’t care about accuracy and worth in standard RDBMS and Data Mining ?  The point of the 3Vs is that they separate Big Data from normal storage and analysis, veracity and value are key to any analysis regardless of data type or size.

Laney's Paper archived at :

No comments:

Post a Comment