Big Data

Technology has entered almost every sphere of our lives — and as a result, data is being produced at a colossal rate. There are now over a billion Internet users, and the majority of the world's web traffic comes from smartphones. At the current pace, around 2.5 quintillion bytes of data are created every day (Figure 3.8 illustrates the many sources contributing to this flood), and the pace keeps rising as the Internet of Things (IoT) continues to grow.

What is Big Data?

This explosion produces data sets of such enormous volume and complexity that they are given their own name: Big Data. The defining problem is that such data cannot be processed and analysed using traditional data processing tools — for two reasons:

1. It is voluminous — simply too large for conventional tools.
2. It is largely unstructured — it does not fit neatly into rows and columns. Examples of this unstructured flow include:
   • social media posts, instant messages and chats
   • photographs shared through various sites
   • tweets and blog articles
   • news items, opinion polls and the comments on them
   • audio and video chats

The challenges Big Data brings

Big Data is not only about size. Working with it raises a whole set of connected challenges:

• integration of data from many sources
• storage
• analysis
• searching
• processing
• transfer
• querying
• visualisation

Why bother? The value hidden inside …