Checking Missing Values

Checking Missing Values in a DataFrame

Real-world data is often incomplete. A student might be absent for a test, a sensor might fail to record a reading, or a survey question might be left blank. In Pandas, such missing data is represented by the special value NaN (Not a Number). Before you can clean or analyse the data, you first need to know where the gaps are.

Pandas provides the isnull() function specifically for this purpose. When called on a DataFrame, it checks every single cell and returns a new DataFrame of the same shape, filled with Boolean values: True where the original cell was missing (NaN), and False where the value was present.

Consider a DataFrame df that stores the marks of four students — Raman, Zuhaire, Ashravy, and Mishti — across four Unit Tests in five subjects. The data for Unit Test 4 of Raman has missing marks in Maths, Science, and English. Calling df.isnull() will produce a table of False values everywhere except for row 3 (Raman's fourth test), where the columns for Maths, Science, and English will show True.

You can also check a single column. For example, df['Science'].isnull() returns a Series where only the row corresponding to Raman's fourth test is True, and every other row is False. This is useful when you want to focus on missing data in one specific attribute.

Checking if Any Value is Missing in a Column

The isnull() function tells you about each individual cell, but often you just want to know: "Does this column have any missing values at all?" For that, you chain the any() function.

df.isnull().any() returns a Series where each column name is paired with a single True or False. In our marks DataFrame, this would show True for Maths, Science, and English (because each has at least one missing mark), and False for Name, UT, S.St, and Hindi.

You can use the same idea on a single column: df['Science'].isnull().any() returns True because the Science column does contain a missing value. In contrast, df['Hindi'].isnull().any() returns False because every Hindi mark is present.

Counting the Number of Missing Values

Knowing that a column has missing values is one thing; knowing how many is another. To get a count of missing values per column, use sum() after isnull(). Since True is treated as 1 and False as 0 in arithmetic, df.isnull().sum() adds up the True values for each column.

For the marks DataFrame, this would produce:

• Name: 0
• UT: 0
• Maths: 1
• Science: 1
• S.St: 0
• Hindi: 0
• Eng: 1

To get the total number of missing values across the entire DataFrame, you call .sum() twice: df.isnull().sum().sum(). This adds up all the column totals, giving the result 3 (one missing mark each in Maths, Science, and English).

A Practical Consequence: Computing Percentages with Missing Data

The textbook illustrates the impact of missing values through two programs. In the first, Raman's Hindi marks are all present (20, 22, 24, 18). His percentage is calculated as (sum of marks) * 100 / (25 * number of tests), which gives 84%.

In the second program, Raman's Maths marks are 22, 21, 14, and NaN. When you call dfMaths.sum(), Pandas automatically treats the NaN as zero for the purpose of addition. The sum becomes 57. The percentage is then 57 * 100 / (25 * 4), which equals 57%. …