Big data and how it’s processed is a fairly new field. A recent survey by IDC, a market intelligence firm, estimates that 1 per cent of the world’s data has been analysed, but the amount of data is thought to be increasing by more than 40 per cent per year.
Data is only as valuable as the insights that can be gleaned from it. About 80 per cent of digital content is considered unstructured and is hard for machines to understand, so the opportunity for countries to establish themselves as experts and corner the market in processing such data is wide open.
Ireland has a number of research and development centres in this area; there are third-level courses in analytics, and the Government has made clear its intention to highlight the advantages Ireland offers. The latest announcement was a €1 million investment in a big data research centre, which was unveiled last month. The Technology Centre in Data Analytics is a collaboration between DIT, UCD and UCC, and is part of a sustained effort to make Ireland top of the market.
A recent survey by Interxion found that in Ireland, the majority of firms were interested in big data for the purpose of data analytics. Although only 7 per cent of those surveyed said big data was a priority for their organisation, this is set to change, with 62 per cent confident that it would be a priority within three years, and 56 per cent within five years. The figures are higher across the European Union as a whole.
So what can it be used for? It could be as simple as helping a telecoms firm to figure out why subscriber numbers are dropping off, ultimately helping them to retain customers. "The data can contain a lot of rubbish – obsolete and trivial – but it also contains highly valuable information," said Colm Murphy, technical director at Espion, an information security company.
On a grander scale, big data can be used for public health projects, helping to pinpoint the spread of illnesses and identify the possibility of an epidemic. That was seen when Google helped to identify the spread of the H1N1 flu virus in real time in 2009, using a model it had devised to track outbreaks of seasonal flu. Using data on terms being input into Google's search engine, the outbreak was tracked in real time.
“Ultimately this unstructured data has value; the problem is the unstructured data is increasing in volume; there is a variety, in other words there is social media and new data sources that don’t fit into the neat, easy to consume structures; and the velocity and frequency at which it is generated. By the time you get a handle on what’s there, new stuff has already been created,” said Murphy.
Espion is an example of how Ireland can become a hub for processing this type of data.
The company has been building up skills in its business over the past number of years that are useful in the analysis of big data. Espion has specialised in the discovery of electronic data, primarily for court cases and other legal actions. This has allowed it to develop a particular set of skills that are almost directly transferable to mining any large set of data for any purpose.
“The technology and skills we have gleaned over the past 12 years in relation to ediscovery now all of a sudden are of interest to many of the same organisations we’ve been working with for non-related business reason,” said Murphy. “The application of these skills is directly transferable to the world of big data.”