As previously mentioned the name
of this blog, difficult data, is down to me misremembering a quote when I set
it up. The quote I was going for was
from Doug Laney in his 2001 paper on problematic data where he set out the 3 Vs
of data that are difficult to handle in traditional database solutions. Laney's paper is archived at his blog and
there is a link at the foot of this post, it's well worth a read, but
essentially it set out the 3Vs that are considered standard frameworks for Big Data
(this was before Big Data was a thing).
3Vs
Volume is the most obvious of the
three and the one that people seem to get hung up on when they are talking
about Big Data (Well big is in the name).
The idea here is that with huge amounts of data the traditional database
architecture has problems coping and by traditional we mean relational. The problem is that this is probably the
least justifiable of the 3 Vs in most Big Data applications. The size of data needed before an enterprise Relational Database Management System
(RDBMS) starts to struggle is way beyond the vast majority of applications, until
the data collection is headed into the Petabyte territory there isn't a huge
justification for moving from a standard database arrangement however the other
2 Vs are often more common and useful.
Velocity, the second of the Vs
looks at the idea of data coming in faster than we would normally deal with,
particularly if this data has limited long term value. As we start to look at Internet of Things
(IoT) applications where we could potentially have hundreds of sensors
collecting real-time data many times a second this becomes apparent. Although the total volume being collected
could easily be well within a normal RDBMS capability having all that data
available quickly and from concurrent sources is quite a challenge, the
problems with sharding RDBMSs are well documented.
The third V is variety. Here we are concerned with rich datasets that
could potentially contain images, sound, video, drawings, tweets, documents
etc. These types of data are very difficult to represent within a relational framework
where every database entry is an atomic intersection of an attribute and a
tuple, even more sophisticated object relations simply allow us to store the
item as an object but still not break it down any further within the database
itself.
4 and 5 Vs and why it misses the point.
Since Laney’s paper we’ve seen
the growth of Big Data which tries to address some of the failings of
traditional RDBMSs and small scale analytics (and replaces them with a few of
its own). Laney’s descriptions have been
largely taken as the definition of suitable areas for implementing Big Data
solutions, any time you have a problem with one or more of the 3 Vs then it’s
time to consider a Big Data implementation.
Of course once a thing gets going we can’t just leave it at that now can
we ? You are only using 3 Vs still ? Psssht!
Marketing departments at certain
companies (I’m looking at you in particular IBM) have expanded the original 3V concept
further into 4 and 5 Vs. Now here the additional
Vs are Veracity (how accurate the data is) and Value (can you actually make
money / value from the analysis). On the
one hand there is no doubt that in order for a Big Data implementation to be
worth doing the additional Vs are necessary.
No one would argue that a successful implementation needs to have data
that is accurate and produces output that reflects reality, and of course any
analysis worth producing should of sufficient quality to act upon, there’s no
point just analysing for analysis sake (even me, a mere academic, understands
that). Where I take issue with IBM et al
here is that they appear to be casting this as a uniquely Big Data thing. Suddenly we don’t care about accuracy and
worth in standard RDBMS and Data Mining ? The point of the 3Vs is that they separate Big Data from normal storage
and analysis, veracity and value are key to any analysis regardless of data
type or size.
Laney's Paper archived at :
No comments:
Post a Comment