Field note · June 25, 2024
The missing values were the finding
18% of salary fields were blank, and the blanks were not random.
Someone on the team was building a salary benchmark from job postings. About 18% had no salary band. The first draft filtered them out and computed averages over the rest.
The blanks are not random. Junior roles hide salary more often than senior ones. Non-US postings hide it far more often than US ones. Both of those correlate with lower pay.
So the average over disclosed salaries is biased upward, and the bias is not small — on our synthetic version of this, data-job-postings, it runs several percent high and worse in some segments.
The check takes one line and I now run it reflexively on any column with meaningful missingness:
df.assign(no_salary=df["salary_min_usd"].isna()) \
.groupby("no_salary")[["seniority", "country", "remote"]] \
.agg(lambda s: s.value_counts(normalize=True).head(2).to_dict())If the two groups look different, the missingness is informative and dropping those rows changes your answer.
What to do about it is harder than detecting it. Options, roughly: report only within strata where disclosure is high; model the missingness explicitly; or state the bias and its direction plainly. We did the third — "this benchmark covers postings that disclose a band, which skews senior and US, so treat it as an upper bound" — because it was honest and took one sentence.
The general point: isna().mean() tells you how much is missing. It tells you nothing about whether that matters. The grouping is where the information is.