Field note · January 31, 2026
2.1% of arr_delay_min is missing
167 blanks in one column, and the one-line check that decides whether dropping them is free.
arr_delay_min on flight-delays is null in 167 of 8,000 rows.
2.1% missing is enough to matter and small enough to ignore, which is the dangerous range. Nothing errors. The aggregate returns. The number is wrong by an amount nobody can see.
The question is never "how much is missing" — that is one line and it tells you nothing. It is whether the rows with a blank differ from the rows without one:
missing = df["arr_delay_min"].isna()
df.groupby(missing)["carrier"] \
.value_counts(normalize=True) \
.unstack(fill_value=0) \
.round(3)If the two rows of that table look alike, dropping the blanks costs you sample size and nothing else. If they look different, the missingness is carrying information, and every average computed over the survivors is biased in a direction you can usually name.
Here the blanks cluster rather than scatter, which means the surviving rows are not a random sample of the column.
"How much is missing" is a completeness metric. "Who is missing" is the finding. The second question costs one more line of code and is the one worth asking.