Q.What are missing values? What are the strategies to handle them?

Missing values, often represented as NaN (Not a Number), None, null, or empty cells, are data points that are not recorded or stored for a given observation in a dataset. They can arise due to various reasons such as data entry errors, equipment malfunction, survey non-response, data corruption, or simply because the information was not applicable or collected. Handling missing values is a crucial step in data preprocessing, as many machine learning algorithms cannot work with incomplete data.

The strategies to handle missing values generally fall into two main categories: Deletion and Imputation.

Strategies to Handle Missing Values

1. Deletion Methods

Deletion methods involve removing the rows or columns that contain missing values.

• Row-wise Deletion (Listwise Deletion)

  This strategy involves removing entire rows (observations) that contain one or more missing values.

  • When to use: It is simple to implement and effective when the number of missing values is small, and the dataset is large.
  • Drawback: It can lead to a significant loss of valuable data, especially if missing values are spread across many rows, potentially introducing bias if the missingness is not random.

• Column-wise Deletion

  This strategy involves removing entire columns (features) that have a high percentage of missing values.

  • When to use: If a particular column has a very large proportion of missing data (e.g., more than 70-80%), it might be better to drop that column entirely, as it may not provide much useful information.
  • Drawback: It results in the loss of an entire feature, which might be important if the missingness could have been handled differently.

2. Imputation Methods

Imputation methods involve filling in the missing values with estimated or substituted values. This approach preserves the dataset size and often retains more information than deletion.

• Mean/Median Imputation (for Numerical Data)

  • Mean Imputation: Missing values in a numerical column are replaced by the mean (average) of the non-missing values in that same column.
  • Median Imputation: Missing values in a numerical column are replaced by the median (middle value) of the non-missing values in that same column.
  • When to use: These are simple and quick methods. Median imputation is preferred over mean imputation when the data contains outliers, as the median is less sensitive to extreme values.
  • Drawback: They do not account for the relationships between features and can reduce the variance of the data, potentially leading to biased estimates.

• Mode Imputation (for Categorical Data)

  Missing values in a categorical column are replaced by the mode (most frequent category) of the non-missing values in that same column.

  • When to use: This is the most common and straightforward method for handling missing values in categorical features.
  • Drawback: It can introduce bias by over-representing the most frequent category.

• Forward Fill / Backward Fill (for Time Series or Ordered Data) …

What Are Missing Values?

Missing values are entries where no data was recorded for a particular row/column combination -- shown in pandas as NaN (Not a Number). They show up for many real reasons: a student was absent for a test, a sensor failed to log a reading, a form field was left blank, or a value genuinely doesn't apply to that record. Left unhandled, missing values break most statistical functions and machine-learning algorithms, which is why Section 3.8 of this chapter treats "detect the gaps, then decide a strategy, then apply it" as a standard workflow.

The Two Families of Strategy

1. Deletion -- remove the incomplete rows or columns.

• Row-wise (dropna()): drop any row containing a missing value. Simple, but every dropped row is lost information -- fine when only a handful of rows are affected, risky when many are.
• Column-wise: drop an entire column if too much of it is missing (rule of thumb: more than 70-80%) -- you lose the whole feature, but keep every row.

2. Imputation -- keep every row, but fill the gap with an estimate.

• Constant fill (fillna(0), fillna(value)): replace with a fixed number -- appropriate when "missing" genuinely means a specific value (e.g. a test not attempted counted as 0 marks).
• Mean/median fill: replace with the column's average -- median is safer when the column has outliers.
• Mode fill: replace with the most frequent value -- the standard choice for categorical columns.
• Forward/backward fill (fillna(method='pad') / fillna(method='bfill')): carry the previous (or next) valid value forward -- natural for ordered data, such as a student's test-to-test performance.
• Interpolation (interpolate()): estimate a value that fits smoothly between the surrounding data points.
• Model-based (KNN, regression): predict the missing value from the OTHER columns -- more accurate, but heavier to set up, and not covered in this chapter.

Seeing the Difference a Strategy Makes, With the Chapter's Own Data

This chapter's case study (Section 3.8) gives a perfect illustration: Raman was absent for Maths, Science and English in Unit Test 4 (S.St and Hindi ARE present that test). Applying three different, equally legitimate strategies to essentially the same problem -- "what percentage did Raman score?" -- gives three different, honest answers:

Strategy 1 -- delete the incomplete row (dropna()):

dfRaman = df[df['Name']=='Raman']
dfRaman.dropna(inplace=True, how='any')   # UT4's row is removed entirely

With UT4 gone, only 3 Unit Tests remain, so Maths' percentage is computed out of 75 (25x3) instead of 100 -- and comes out to 76.0%.

Strategy 2 -- fill the gap with zero (fillna(0)):

dfRaman = df.loc[df['Name']=='Raman']
(row, col) = dfRaman.shape            # row = 4 (all 4 Unit Tests, NaN included)
dfScience = dfRaman.loc[:, 'Science']
dfFillZeroScience = dfScience.fillna(0)
print(dfFillZeroScience.sum() * 100 / (25 * row), "%")

Output:

60.0 %
``` …