Big ideas are needed to fully harness the power of big data

That’s Maths: More sophisticated methods are required to extract value from the big-data avalanche

Globally, something like five exabytes of data are created every day. That is about a million million million words. Image: iStock

Globally, something like five exabytes of data are created every day. That is about a million million million words. Image: iStock

 

The world is awash with data.. With mobile devices and internet connections, data capture is simple and powerful computers make the analysis of “big data” feasible. But there are challenges: many data sets are too large and too complex to be analysed or understood using traditional data-processing methods. Our current armoury of analysis techniques is inadequate and new mathematical methods are needed.

Globally, something like five exabytes of data are created every day. That is about a million million million words. The data comes from a multitude of sources. Among these are internet traffic, phone calls, education, medical and health records, court reports, genome sequences, astrophysical observations, stock-market movements and social networks. On Twitter about 6,000 tweets are sent every second, which means 500 million per day and about 200 billion per year.

Into the matrix

Small data sets can be organised into a matrix using a simple spreadsheet. For big data, processing is beyond human capacity and more sophisticated methods are required: new, more efficient forms of processing are needed to extract value from the data. Big data presents huge management tasks: data capture, verification, storage, sorting, analysis, visualisation and presentation.

Big data is noisy, unstructured and ever-changing. Real-time analysis requires hugely parallel processing with software running simultaneously on thousands of processors. Brute-force analysis alone is ineffective, and modern computer architectures require innovative algorithms to exploit their power. Specialised software tools are available, such as Hadoop, a software framework for distributed storage and processing of very large data sets on computer clusters, and Presto, a system developed by Facebook for running interactive analytic queries.

Unlocking the information from large data sets yields understanding and enables predictions about future trends. Big data analysis can reveal new links and relationships. Large companies – eBay, Amazon, Netflix, Facebook and Google – are engaged in analysis of customer preferences: patterns of purchases enable them to recommend products that a customer is likely to buy. Other applications include insurance-fraud detection, flight analysis and medical diagnosis and prognosis.

Humans are poor at heavy quantitative analysis but brilliant at pattern recognition. For example, we might be able to recall a face seen briefly years ago. So far, machines cannot match us but, as large data analysis progresses and new techniques are developed, substantial advances can be expected. Millions of images can be input to deep-learning algorithms to train them to recognise patterns.

Many human activities are organised in networks, which can be modelled using graph theory. A graph is just a collection of nodes linked by edges, like an electric- circuit diagram or a railway map. The branch of mathematics that deals with connectivity and continuity is called topology, and it includes graph theory. Topological data analysis provides a way of generating structured data sets from unstructured, chaotic data. The structured data can then be processed using algorithms.

Often, data is represented by points in a high-dimensional space – difficult to visualise but amenable to algebraic manipulation. Large multidimensional data sets can be reduced to a greatly compressed state using methods such as singular value decomposition, where the essential information-bearing components are isolated and the rest dumped.

There is an acute shortage of experts in data analysis in many industrial sectors, including health, finance, climate science, pharmaceuticals and online services. Several universities, UCD included, offer postgraduate programmes in data analytics. With so many open problems, this is a promising field for young people.

  • Peter Lynch is emeritus professor at the school of mathematics and statistics, University College Dublin. He blogs at thatsmaths.com
The Irish Times Logo
Commenting on The Irish Times has changed. To comment you must now be an Irish Times subscriber.
SUBSCRIBE
GO BACK
Error Image
The account details entered are not currently associated with an Irish Times subscription. Please subscribe to sign in to comment.
Comment Sign In

Forgot password?
The Irish Times Logo
Thank you
You should receive instructions for resetting your password. When you have reset your password, you can Sign In.
The Irish Times Logo
Please choose a screen name. This name will appear beside any comments you post. Your screen name should follow the standards set out in our community standards.
Screen Name Selection

Hello

Please choose a screen name. This name will appear beside any comments you post. Your screen name should follow the standards set out in our community standards.

The Irish Times Logo
Commenting on The Irish Times has changed. To comment you must now be an Irish Times subscriber.
SUBSCRIBE
Forgot Password
Please enter your email address so we can send you a link to reset your password.

Sign In

Your Comments
We reserve the right to remove any content at any time from this Community, including without limitation if it violates the Community Standards. We ask that you report content that you in good faith believe violates the above rules by clicking the Flag link next to the offending comment or by filling out this form. New comments are only accepted for 3 days from the date of publication.