Field note · April 21, 2026

Check the blanks before you drop them

89 blanks in one column, and the one-line check that decides whether dropping them is free.

1 min read ·Statistics ·quality statistics

The default move on a column with blanks is to filter them out. On week12_score in clinical-trial that would quietly change the answer.

9.9% missing is enough to matter and small enough to ignore, which is the dangerous range. Nothing errors. The aggregate returns. The number is wrong by an amount nobody can see.

The question is never "how much is missing" — that is one line and it tells you nothing. It is whether the rows with a blank differ from the rows without one:

python
missing = df["week12_score"].isna()

df.groupby(missing)["arm"] \
  .value_counts(normalize=True) \
  .unstack(fill_value=0) \
  .round(3)

If the two rows of that table look alike, dropping the blanks costs you sample size and nothing else. If they look different, the missingness is carrying information, and every average computed over the survivors is biased in a direction you can usually name.

Here the blanks cluster rather than scatter, which means the surviving rows are not a random sample of the column.

"How much is missing" is a completeness metric. "Who is missing" is the finding. The second question costs one more line of code and is the one worth asking.