Data Collection

Before any data can be processed, it must first exist in our hands — so the very first stage of working with data is data collection. Once collected, the data can be stored in a file or a database for later use. Data collection means either identifying data that is already available or collecting it from the appropriate sources.

Three collection scenarios

The situation on the ground decides how collection actually happens. Take a grocery store whose sales data we want:

• The data exists only on paper. The shopkeeper keeps sales records in a diary or register. Here the data must be entered into a digital format — for example, into a spreadsheet.
• The data is already digital. The records are already in a digital form, say a CSV (comma separated values) file, and can be used as they are.
• No data has been recorded at all. The shopkeeper has kept no records in either form, but wants software developed for maintaining sales data and accounts. Such software may be built using a programming language such as Python, which can store and retrieve data from a CSV file or from a database management system like MySQL.

These three cases cover most real projects: digitise paper records, reuse existing digital data, or build a system that captures data from now on.

Data is being generated everywhere, all the time

Data are continuously being generated at different sources. Every interaction we have with the digital medium adds to a huge and growing volume of data. Organisations deliberately collect this flow because analysing it improves their decisions:

• Hospitals collect data about patients in order to improve their services.
• Shopping malls collect data about the items people purchase. Suppose analysis of this data shows that bedsheets and groceries are frequently bought together — the owner may then decide to display bedsheets near the grocery section to lift sales.
• Political analysts examine the posts and messages on a social media platform to gauge public opinion before an election. …