Handling Missing Values

Handling Missing Values

A DataFrame can contain many rows, where each row holds values for different columns (attributes). When a value for a particular column is absent, it is called a missing value. In pandas, a missing value is represented as NaN (Not a Number).

In real-world datasets, missing values are common. They can arise for several reasons. Data may not have been collected properly — for example, some people might skip fields in a survey. Sometimes, an attribute is simply not relevant to a particular person. If someone is unemployed, the "salary" attribute would be irrelevant and likely left blank.

Missing values cause serious problems during data analysis and must be handled carefully. The textbook explains two common strategies for dealing with them:

• Drop the object (row) that has missing values.
• Fill or estimate the missing value.

A Case Study Example

Consider the earlier case study of students and their unit test marks. Now, the students have appeared for Unit Test 4 as well. However, Raman could not take the Science, Maths, and English tests for Unit Test 4, and there is no possibility of a re-test. So, his marks for those subjects in Unit Test 4 are missing.

The dataset after Unit Test 4 is shown in Table 3.2. Notice that for Raman, the columns 'Science', 'Maths', and 'English' have missing values in the row for Unit Test 4.

To calculate the final result, teachers need to submit the percentage of marks for all students. Each teacher handles Raman's missing values differently:

• The Maths teacher decides to compute marks based on only the three tests Raman did take, and then finds the percentage out of a total of 75 marks (instead of 100). In effect, she drops the marks of Unit Test 4. …