Field note · June 5, 2024
Reading week12_score before trusting it in clinical-trial
900 rows, 434 distinct values, and the one fact about week12_score that changes how you query it.
Profiling week12_score on clinical-trial before anyone builds anything on top of it.
900 rows, 89 nulls (9.9%), 434 distinct values. Score at week 12. Empty if the subject dropped out.
The five-number version: min 7.00, p10 35.10, median 52.50, p90 70.90, max 98.40. The mean is 52.80.
The p10 to p90 band — 35.10 to 70.90 — is where ordinary rows live, and it is the pair worth quoting when somebody asks what to expect. Min and max describe the two strangest rows in the table and nothing else; they are useful for spotting impossible values and misleading for everything else.
89 rows have no value at all here, which is 9.9% of the table. That is enough to move an aggregate and small enough that nobody notices it doing so — every average above is computed over 811 rows, not 900, and the two denominators produce different answers.
select
count(*) as rows,
count(week12_score) as present,
count(*) - count(week12_score) as nulls,
count(distinct week12_score) as distinct_values
from clinical_trial;Nothing dramatic: the mean and median are within 0.6% of each other, so an average is a fair summary. That is worth confirming rather than assuming — it is not true of most money columns.
Where this matters: a symmetric column is one you can average, threshold and chart without hedging, which makes it unusually cheap to work with. Knowing which of your columns are like this and which are not is most of knowing when to be careful.
The grain is one row per enrolled subject, which is the context every one of those numbers depends on. None of them survive a change of grain, which is why "profile the column" and "profile the table" are the same job. Full schema, and the CSV, on the dataset page.