One of the Vs of big data is the idea of dealing with data variety, whilst
volume and velocity are fairly straight forward to understand variety can be a
bit trickier to understand. The idea behind variety is the concept of structured
and unstructured data (and occasionally semi structured). Structured data is the
traditional single atomic values we see in normal database usage, the idea here,
according to standard relational theory, is that the data value is as far down
as we can examine. An example of this type of data would be something like an
address, Joe Bloggs lives at 33 Madeup Road, NotRealton, Fantasyshire, XX9 9XX.
To store Joe's information in a standard RDBMS we would split the information up
into attributes, lets suppose we had settled on name and address. Joe's address
is a string (33 Makeup Road, NotRealton, Fantasyshire, XX9 9XX) it is stored as
the entire string, an atomic value that we can't split in the RDBMS, we can do
tricks like using SQLs "contains" keyword but fundamentally we can't separate his
postcode from his county, if we wish to do this we would have to redesign the
database schema to store different parts of the address in different attributes.
Now this is perfectly feasable and sensible to do and we've been doing this
since the 70's, in many instances (the vast majority) RDBMSs make absoulte
sense, DBAs are worth their weight in gold and SQL is the language of the Gods (
occasionally mean, spiteful Gods but still...) however...
What happens in this new world of Big Data ? Now someone from marketing appears
and says "Hey we've been engaging our customers on social media and we've got
responses on twitter along with facebook posts, vines, instagrams and emails it'd
be great if you could do something with that." Ok what do we do now (quietly
murdering the marketing department isn't an option as they'll just replace them
anyway)? This type of data is generally thought of as unstructured, it doesn't
fit nicely into our RDBMS model, sure we can store it but we'd just be storing
the tweet or post as an object, it would be extremely difficult to make any use
of it beyond storage and retrieval. If we are going to be pendantic (and who
doesn't like being pendantic ?) this data isn't really unstructured it's semi
structured, we know what the format of a tweet is (140 char limit may include
links or images) but there is such variety in what it might look like and the
information it could contain that we can't sensibly organise a set of attributes
like we can for an address. If the data was truely unstructured then we
wouldn't know anything about it at all and there isn't really a lot we'll be
able to do with that (not without a whole heap of natural language, image
recognition etc. jut to figure out what we have.). We occasionally come across
examples where someone tries to breakdown data into semi and unstructured but in
most instances this type of data (along with video, sound, documents etc.) are
considered to be unstructured.
By storing this type of data in a non tradional datastore we can get access at a
deeper gradualarity. The type of store needed will greatly depend on the type of
data being collected, but NoSQL databases vary in a huge variety from graph to
JSON to pretty much anything we might want to store. Of course when the marketing
department comes back in a few months time and says thanks for the analysis on the
data we sent but "can we do this in real time as responses to tweets and instagrams"
we have a whole new set of problems, but that's a different post (and not just being
suspicous because marketing said thanks).