Big Data

"Big data is a term for data sets that are so large or complex that traditional data processing applications are inadequate to deal with them. Challenges include analysis, capture, data curation, search, sharing, storage, transfer, visualization, querying, updating and information privacy. The term 'big data' often refers simply to the use of predictive analytics, user behavior analytics, or certain other advanced data analytics methods that extract value from data, and seldom to a particular size of data set."
— Wikipedia

This article, Inside American Express' Big Data Journey, explores what is needed to make the shift to using big data effectively.

You need to be prepared for a journey. The changes needed won't happen overnight.

You need buy-in from the top of your organisation. Without that, you won't get the mandate for the investment and the culture shift that is required.

At the moment, big data technologies are still being developed, so you need to adapt to using new tools, and be prepared to upgrade them regularly.

You also need to recruit and retain new talent. There are not that many people around with experience of big data, so you may need to get creative and be flexible. Once you have recruited the talent, you need to work out how to hang on to them, as they are likely to want to move on to the next challenge.

You need to be prepared for a process of continuous improvement and iterative learning. This applies to both development and marketing. This sounds like Agile to me, which means the whole company will need to adapt to working in an Agile way, becoming tolerant of a trial-and-error approach.

American Express also made the use of big data available to more decision-makers within the company, empowering people to "act locally", enabling new ways for customers to use their services. It also allowed them to get rid of obsolete services, and reduce levels of fraud, saving huge amounts of money and time. The benefits from fraud improvement alone have paid for their investment in big data.

Where to start?

Before you can get to Big Data - data sets that are so large that they need powerful tools to analyse them, and better tools for visualising them as networks or graphs or interactive infographics - first you need data interoperability.

Before you can access the enormous goldmine of Big Data to analyse requirements and trends in your sector, you need to be able to exchange data with your partners. Of course, HE is already submitting large data sets to HESA and HEFCE.

This means that you either need to embrace interoperable data standards (a time-consuming and thankless task), or you need a tool that will convert data from one format to another.

The Next Big Thing

According to this article by Matt Turck, the next big thing after big data will be using artificial intelligence (AI) to analyse the enormous datasets and add value and business intelligence to your business processes.

Don't Get Left Behind

In the brave new world of Big Data and AI, the biggest players will be the organisations that are using Big Data natively and naturally to underpin all their processes.

In this article, Big Data Overload: Why Most Companies Can't Deal With The Data Explosion, Bernard Marr says that "it's all very well collecting data, but what are you going to do with it once you've got it?" If you can't analyse your data, there's not much point in having it.

That's where smart data comes in.

Marr writes: "Without a smart plan of action to use the data to produce business insights, the data itself becomes a white elephant — expensive and useless."

That's why you need to work out what you want to know, and ask the right questions, before you start collecting data.

A good place to start is the classic set of questions, "Who, What, Why, Where, How?"

This data gives you the potential for visibility which, in turn, allows you to make informed decisions based on actual events - and use these historical events to predict future occurrences.

If the data's there, you've got to bring it together. Like the article says: you've got to use it, or you'll lose it, and your organisation's ability to act on it in a timely way.

Using Big Data for Good

Of course, Big Data can be used for nefarious purposes such as working out Banksy's real identity, but it can also be used for good purposes.

In this excellent article on data and gender, Melinda Gates points out that many women disappear from statistics and from the records of governments and quangos, simply because their existence is never recorded.

Presumably the 200 Chibok girls who were abducted by Boko Haram could have been rescued more easily if they had been better documented - to date, only one of them has been rescued.

It would be easier to prevent female genital mutilation, forced marriages, child marriages, and other horrors, if the data on women existed.

The Bill and Melinda Gates Foundation is going to invest $80 million over the next three years to help improve the way data is collected   and used.

This will provide a much more detailed picture of the challenges women and girls face, and what can be done to overcome them.