Wednesday, 4 January 2017

Katana ROS drivers on Kinetic Kame.


NB. Editted 25/01/17 for a minor typo (Word capitalising a word it shouldn't have) and one step that needed to be moved (installing the rosdep dependencies needs to be done AFTER removing the unneeded/unsupported packages.)

Just before Christmas I decided to try out the current ROS version (Kinetic Kame), here at UCLan we've been running with Indigo for quite a while to avoid compatibility issues but I thought I'd give it a go.

Now installing the Katana drivers has never been straight forward, I've never got the instructions on the University of Osnabrück page to work and always ended up trawling through missing dependencies but they've been updated for Jade so how painful could it really be ?

Well...  teeth have been pulled, chickens sacrificed and elder gods appeased, and I present to anyone interested my install guide.

Bits in bold need typing in.

Usual disclaimers, it works for me, may not for you, may have some redundant steps and packages that aren't fully needed, don't copy and paste it as it was written up in Word as an internal doc, may cause depression and occasional bleeding.  Worked at the end of 2016.
If anyone has a simpler way please (for the love of sanity) say so in the comments.

ROS / Katana Stack Install Guide.
This presumes a standard Ubuntu install to begin with.
This document will document install of Kinetic Kame and the Katana drivers from University of Osnabrück.

Standard warning : Do not copy and paste command line instructions from this document.  This document was written in MS Word and uses a different character set than the standard ANSI.
Following instructions from standard ROS install from wiki.ros.org/kinetic/Installation/Ubuntu:
First set ROS repositories. 

sudo sh -c ‘echo “deb http://packages .ros.org/ros/ubuntu $(lsb_release -sc) main” > /etc/apt/sources.list.d/ros-lastest.list’
Get the appropriate keys.
sudo apt-key adv --keyserver hkp://ha.pool.sks-keyservers.net:80 -- recv-key 0xB01FA116
Update the Ubuntu package list.
sudo apt-get update
Finally install the full standard ROS setup.
sudo apt-get install ros-kinetic-desktop-full
Next we need to initialise rosdep.
sudo rosdep init
rosdep update
Setting up bash for ROS use.
echo “source /opt/ros/kinetic/setup.bash” >> ~/.bashrc
source ~/.bashrc
Check that the lines above entered the line below into your ~/.bashrc file (it occasionally doesn’t seem to work) if it hasn’t (should be the last line) edit it manually.
source /opt/ros/kinetic/setup.bash
Installing rosinstall.
sudo apt-get install python-rosinstall
ROS should now be installed and setup.
Next installing Katana drivers.
Make a directory for the drivers
mkdir -p ~/ros-kinetic/src/
Move to that directory.
cd ~/ros-kinetic/src
Here is where we deviate from the on-line install instructions, there are a few things to install separately.
sudo apt-get install build-essential
sudo apt-get install ros-kinetic-ivcon
sudo apt-get install ros-kinetic-convex-decomposition
sudo apt-get install ros-kinetic-navigation
sudo apt-get install python-catkin-tools
sudo apt-get install ros-kinetic-gazebo-ros-control
sudo apt-get install ros-kinetic-ros-control
sudo apt-get install ros-kinetic-ros-controllers
sudo apt-get install ros-kinetic-sound-play
sudo apt-get install ros-kinetic-joystick-drivers
sudo add-apt-repository ppa:kbs/kbs
sudo apt-get update
sudo apt-get install libepos2-dev
sudo apt-get install ros-kinetic-slam-gmapping
sudo apt-get install ros-kinetic-frontier-exploration
sudo apt-get install libarmadillo-dev
sudo apt-get install ros-kinetic-moveit
sudo apt-get install ros-kinetic-moveit-python

Next download the Katana University of Osnabruck stack.
wstool init
wstool merge https://raw.githubusercontent.com/uos/uos_rosinstalls/master/uos-public-indigo-wet.rosinstall
wstool update
Next install dependencies.
rosdep update


The UoS stack contains driver we don’t need that have dependencies that haven’t been updated for ROS Kinetic yet so at this point we need to delete some directories :
cd  ~/ros-kinetic/src
sudo rm -rf calvin_robot
sudo rm -rf calvin_navigation
sudo rm -rf kurt_driver
sudo rm -rf robot_self_filter
sudo rm -rf uos_rotunit
There is now a change needed to one of the packages :
cd ~/ros-kinetic/src/uos_tools/rotunit_snapshotter
Inside this directory there is a CMakeList.txt file, open it in a text editor using sudo and add the following line to the very end of it :
include_directories(${catkin_INCLUDE_DIRS})

Kinetic has moved to a minimum C++11 standard which seems to cause some issues, we need to make a change to one of the ROS header files :
cd /opt/ros/kinetic/include /moveit/macros/
sudo edit the file declare_ptr.h
In the file where the anywhere the code says :
typedef std::shared_ptr…
Replace std:: with boost::

Install the rosdep dependencies.

rosdep install --from-paths ./ -i -y --rosdistro kinetic


Next build the Katana packages
cd ~/ros-kinetic/src
catkin_init_workspace
cd ..
catkin_make
. ~/ros-kinetic/devel/setup.bash

Setting up the user to use USB port.
Add user to dialout group (where <username> is your username:
sudo gpasswd -a <username> dialout
The following needs to be done everytime the machine is booted so it is worth putting in a bash script :
cd /dev
sudo rm ttyS0
sudo ln /dev/ttyUSB0 /dev/ttyS0
sudo chmod 666 /dev/ttyS0
sudo chown root:dialout /dev/ttyS0

Add the katana type into the .bashrc shell script with the following line :
export KATANA_TYPE=”katana_300_6m180”



Friday, 20 May 2016

Structured and Unstructured Data

One of the Vs of big data is the idea of dealing with data variety, whilst
volume and velocity are fairly straight forward to understand variety can be a
bit trickier to understand. The idea behind variety is the concept of structured
and unstructured data (and occasionally semi structured).  Structured data is the
traditional single atomic values we see in normal database usage, the idea here,
according to standard relational theory, is that the data value is as far down
as we can examine.  An example of this type of data would be something like an
address, Joe Bloggs lives at 33 Madeup Road, NotRealton, Fantasyshire, XX9 9XX.
To store Joe's information in a standard RDBMS we would split the information up
into attributes, lets suppose we had settled on name and address.  Joe's address
is a string (33 Makeup Road, NotRealton, Fantasyshire, XX9 9XX) it is stored as
the entire string, an atomic value that we can't split in the RDBMS, we can do
tricks like using SQLs "contains" keyword but fundamentally we can't separate his
postcode from his county, if we wish to do this we would have to redesign the
database schema to store different parts of the address in different attributes.
Now this is perfectly feasable and sensible to do and we've been doing this
since the 70's, in many instances (the vast majority) RDBMSs make absoulte
sense, DBAs are worth their weight in gold and SQL is the language of the Gods (
occasionally mean, spiteful Gods but still...) however...

What happens in this new world of Big Data ?  Now someone from marketing appears
and says "Hey we've been engaging our customers on social media and we've got
responses on twitter along with facebook posts, vines, instagrams and emails it'd
be great if you could do something with that." Ok what do we do now (quietly
murdering the marketing department isn't an option as they'll just replace them
anyway)?  This type of data is generally thought of as unstructured, it doesn't
fit nicely into our RDBMS model, sure we can store it but we'd just be storing
the tweet or post as an object, it would be extremely difficult to make any use
of it beyond storage and retrieval.  If we are going to be pendantic (and who
doesn't like being pendantic ?) this data isn't really unstructured it's semi
structured, we know what the format of a tweet is (140 char limit may include
links or images) but there is such variety in what it might look like and the
information it could contain that we can't sensibly organise a set of attributes
like we can for an address.  If the data was truely unstructured then we
wouldn't know anything about it at all and there isn't really a lot we'll be
able to do with that (not without a whole heap of natural language, image
recognition etc. jut to figure out what we have.).  We occasionally come across
examples where someone tries to breakdown data into semi and unstructured but in
most instances this type of data (along with video, sound, documents etc.) are
considered to be unstructured.

By storing this type of data in a non tradional datastore we can get access at a
deeper gradualarity.  The type of store needed will greatly depend on the type of
data being collected, but NoSQL databases vary in a huge variety from graph to
JSON to pretty much anything we might want to store.  Of course when the marketing
department comes back in a few months time and says thanks for the analysis on the
data we sent but "can we do this in real time as responses to tweets and instagrams"
we have a whole new set of problems, but that's a different post (and not just being
suspicous because marketing said thanks).

Tuesday, 29 March 2016

Difficult / Problematic Data and the 3 Vs



As previously mentioned the name of this blog, difficult data, is down to me misremembering a quote when I set it up.  The quote I was going for was from Doug Laney in his 2001 paper on problematic data where he set out the 3 Vs of data that are difficult to handle in traditional database solutions.  Laney's paper is archived at his blog and there is a link at the foot of this post, it's well worth a read, but essentially it set out the 3Vs that are considered standard frameworks for Big Data (this was before Big Data was a thing).
                                                                                                                           
3Vs
Volume is the most obvious of the three and the one that people seem to get hung up on when they are talking about Big Data (Well big is in the name).  The idea here is that with huge amounts of data the traditional database architecture has problems coping and by traditional we mean relational.  The problem is that this is probably the least justifiable of the 3 Vs in most Big Data applications.  The size of data needed before an enterprise Relational Database Management System (RDBMS) starts to struggle is way beyond the vast majority of applications, until the data collection is headed into the Petabyte territory there isn't a huge justification for moving from a standard database arrangement however the other 2 Vs are often more common and useful.

Velocity, the second of the Vs looks at the idea of data coming in faster than we would normally deal with, particularly if this data has limited long term value.  As we start to look at Internet of Things (IoT) applications where we could potentially have hundreds of sensors collecting real-time data many times a second this becomes apparent.  Although the total volume being collected could easily be well within a normal RDBMS capability having all that data available quickly and from concurrent sources is quite a challenge, the problems with sharding RDBMSs are well documented.

The third V is variety.  Here we are concerned with rich datasets that could potentially contain images, sound, video, drawings, tweets, documents etc. These types of data are very difficult to represent within a relational framework where every database entry is an atomic intersection of an attribute and a tuple, even more sophisticated object relations simply allow us to store the item as an object but still not break it down any further within the database itself.

4 and 5 Vs and why it misses the point.
Since Laney’s paper we’ve seen the growth of Big Data which tries to address some of the failings of traditional RDBMSs and small scale analytics (and replaces them with a few of its own).  Laney’s descriptions have been largely taken as the definition of suitable areas for implementing Big Data solutions, any time you have a problem with one or more of the 3 Vs then it’s time to consider a Big Data implementation.  Of course once a thing gets going we can’t just leave it at that now can we ?  You are only using 3 Vs still ? Psssht!

Marketing departments at certain companies (I’m looking at you in particular IBM) have expanded the original 3V concept further into 4 and 5 Vs.  Now here the additional Vs are Veracity (how accurate the data is) and Value (can you actually make money / value from the analysis).  On the one hand there is no doubt that in order for a Big Data implementation to be worth doing the additional Vs are necessary.  No one would argue that a successful implementation needs to have data that is accurate and produces output that reflects reality, and of course any analysis worth producing should of sufficient quality to act upon, there’s no point just analysing for analysis sake (even me, a mere academic, understands that).  Where I take issue with IBM et al here is that they appear to be casting this as a uniquely Big Data thing.  Suddenly we don’t care about accuracy and worth in standard RDBMS and Data Mining ?  The point of the 3Vs is that they separate Big Data from normal storage and analysis, veracity and value are key to any analysis regardless of data type or size.

Laney's Paper archived at :

Tuesday, 16 February 2016

First Post - Intro

Ah the tricky first post.  Ok probably worth mentioning who I am, what I do and what this blog thing is about (it's about 4 years late, aha, ha, no really).  My name is Carl Berry, I'm a lecturer in Robotic Engineering at UCLan (University of Central Lancashire).  Although I'm in the School of Engineering I'm a Computer Scientist really, well Ok technically my undergraduate degree was in Applied Statistics and Computing from Liverpool John Moores and I got My M.Sc. in Information Systems Engineering from UMIST (well that ages me :) ).  I've worked as a research assistant and lecturer at the University of Salford and Manchester Metropolitan University and my main topics of teaching were programming and databases.  Ok, so much for the past, right now I teach robotics, software development and machine learning.  If I wasn't so skeptical about buzzword labels I'd probably say I was a data scientist with robotic leanings.

Woah (I hear nobody say) where does the data scientist label come in ?  Well that is what this is all about.  I'm attached to the Advanced Digital Manufacturing Technologies (ADMT) research group at the Burnley UCLan campus, we recently completed a european funded project called digitme (www.digitme.co.uk) that was concerned with helping small / medium manfacturing companies with data related issues.  My research (and one of the likely offerings in the next project) is on using Big Data techniques for manufacturing processes.  So that's what this blog will be about Big Data, IOT, AI, Robotics and general data related stuff.  There's a very good chance I'll get over excited about something else and go off on a tangent as well (ever sat in one of my lectures ? :) ).

Anyway usual caveats apply, the thoughts expressed here are the ones travelling through my head at the time of writing and do not necessarily reflect those of my employer, my thoughts 12 months down the line or reality in general.  If I get around to posting any code on here it will be of the "works for me" variety and I don't guarantee it will work for you or be safe to use or be anything you'd want to use for sanitys reasons but you are welcome to do so all the same.

I'm also on twitter (God help me) at carlandrewberry. 

Next time : Why is the blog called difficult data ?  (short answer, because I miss remembered a quote)