Big data - the next big thing in technology

THE TECH industry is full of buzz words. While companies may have struggled to get to grips with the cloud in recent years, the latest to hit the market is “big data”.

But what exactly does it mean? According to IBM, 90 per cent of the data in the world today has been created in the past two years.

The explosion in popularity of smartphones has helped fuel this growth, with sites and services such as Facebook, LinkedIn and Twitter helping to create more and more online data.

At the end of June, Facebook’s user base stood at 955 million active monthly users, with 552 million active daily users. Twitter, meanwhile, is generating about 400 million updates every day.

Companies gather data about their customers. Millions of sensors compile information every day. Add into all that the growth in ecommerce, which generates purchase records, digital images, IT logs, GPS signal data and videos, and you aren’t even coming close to the concept that is big data. It’s a massive, multistructured pool of information that could be invaluable, if you know what to do with it.

Ultimately, it differs from the traditional type of “structured” data that we have become used to in the past, which includes everything from bank customer records and financial information to invoices and billing data.

“The world of the iPhone, Twitter, Facebook, YouTube, has created a new data type, a lot of unstructured data,” says EMC’s Jason Ward.

“A lot of these data types are difficult to manage, monitor, model and analyse in the way we used to analyse data.

“When you marry unstructured data and data of old, you get the concept of big data.”

IBM has previously expressed the belief that the “winners” of the big data era will be those who drive innovation and become more competitive by using their “information assets”. In other words, making sense of big data is key for companies.

Quite simply, big data can be used to grow your business, extracting insights into your customers and their behaviour, predicting trends in the market that you can take advantage of, and using real time information from sources such as Twitter to reach out to customers and ensure that they are satisfied by the products and services produced by companies.

It can be used across a range of sectors – from healthcare and telecoms to energy firms and law enforcement. The information gathered through big data can be used to cut down on fraud, enable the introduction of smart metering to help control energy bills and foster better customer relationships.

According to research firm IDC, big data technology and services is a growing market, expected to reach $16.9 billion by 2015, increasing at a rate of almost 40 per cent annually.

Companies such as IBM and EMC are turning to big data as the next big thing in the technology industry, with IBM earlier this year announcing the purchase of software firm Vivisimo.

Intel, Dell, HP and other multinationals are also getting in on the act. And Ireland has the opportunity to position itself as a major player – if we move fast.

Ward says there is now a chance for Ireland to move into the area of analytics. Creating tools that layer on top of big data platforms that create models and trending around customer patterns.

“Big data is increasingly becoming an important asset to commercialise. Lots of organisations, be they large or small, are now looking at ways and are open to ideas through which they can capitalise on the wealth of the real-time data that they are generating,” he says.

New companies are springing up to create and commercialise data models around different types of information, for example creating apps and analytic models to look at customer sentiment and churn.

Using social media sites, retailers could ascertain how likely customers are to move to competitors, allowing them to accurately predict customer churn.

However, to do that, companies need tools to analyse the data.

There’s a big gap in the market around applications that provide these types of services, says Ward, and Irish firms could fill it.

“If Ireland can embrace big data and help companies monetise the data that they have, there’s great opportunities to create new jobs,” he says.

“There is a window of opportunity here. Companies are going to look to where the applications and the skills are to deliver the benefits around commercialising this data.”

The key thing is to provide the skills needed to build up a big data industry in Ireland – that means people with mathematically-led skills such as statisticians. And creating these skills starts early, with the need for a curriculum stream focused on big data.

Colleges have already begun working with the industry to identify what the education system can do to prepare graduates for the world of big data, but Ward says students at all levels need to be prepared in this area.

“It’s a different area within the curriculum but it’s one that does need to be expanded upon,” he says.

That involves retraining and introducing graduates to big data science, and augmenting that with making secondary level students aware of the opportunities that are out there.

It’s a global issue, it seems. A recent report from McKinsey estimated that there is a shortfall of 190,000 data scientists globally.

“No one has really stolen the march. There’s nowhere you can go today and pull out 1,000 data science graduates,” says Ward.

There is another element to big data that could benefit Irish companies – security.

According to a recent security survey, a significant number of Irish firms are not taking the threat of online crime and fraud seriously. A recent survey on such crime carried out by Deloitte in association with EMC, found that although almost a third of those surveyed said they had experienced between one and five security breaches in the past year, fewer than half of firms said it was a priority in terms of risk to the organisation.

According to EMC’s security division RSA, big data could be used effectively to help cut down on fraud and mitigate the effect of such malicious attacks.

Ward says the security models companies previously relied on are inadequate. Companies need to turn to big data and its advances to aid quicker reactions to threats and eliminate any blind spots for firms.

The new infrastructure allows organisations to take an entire view of their IT set-up and see what is happening that is out of the ordinary.

“Once you’re able to monitor and track all traffic within your business, it’s easier to ascertain certain patterns or trends,” says Ward. “Before, the infrastructure to analyse that amount of data was very costly.”

Several attacks have been made on high profile firms in recent months, with LinkedIn, and Last.fmusers among those who have had password data compromised.

Although it’s difficult to prevent malicious attacks – Ward says it’s not a matter of if an organisation will be targeted, but when – finding out quickly that such an intrusion has occurred is important.

Equally important, however, is having backup from the board and senior management. RSA recommends establishing a board risk committee, while also reviewing top-level policies to build up a culture of strong IT security.

Combined with the benefits big data can provide, it will contribute to ensuring security professionals “come armed to fight a battle they can win”, says Ward.

Big data can be used to grow your business, extracting insights into your customers and their behaviour, predicting trends in the market that you can take advantage of