The first hour with an unfamiliar dataset

Someone hands you a dataset. Do not start answering the question. Spend an hour finding out what you have, in this order. Every step has caught something real for me at some point.

1. Shape and grain#

python

df.shape                    # how much, how wide
df.head(20)                 # what does a row look like
df.dtypes                   # what did the reader guess

Then the question from module one: what does one row represent? Test it rather than assume it.

python

# Is the column I think is the key actually unique?
df["order_id"].is_unique
df.duplicated(subset=["order_id", "line_no"]).sum()

If the key is not unique, everything downstream is at risk and you have learned the most important fact about this dataset in ninety seconds.

2. Missingness, and whether it is random#

python

missing = df.isna().mean().sort_values(ascending=False)
print(missing[missing > 0])

Then the question that matters far more than the rate: is the missingness related to anything?

python

# Are rows with a missing salary different from rows without one?
df.assign(no_salary=df["salary_min_usd"].isna()) \
  .groupby("no_salary")[["seniority", "country", "remote"]] \
  .agg(lambda s: s.value_counts(normalize=True).head(2).to_dict())

In data-job-postings the answer is emphatically yes: junior and non-US postings hide salary far more often. An average over disclosed salaries is therefore biased upward, and no amount of careful arithmetic downstream fixes that. This single check is the difference between an analysis and a misleading one.

Watch for sentinel nulls too. -999, 0, 1900-01-01, "N/A", "unknown", empty string. These are nulls in disguise, and isna() will not find them.

python

for col in df.select_dtypes("object"):
    suspicious = df[col].isin(["", "NULL", "N/A", "n/a", "-", "none", "unknown"]).sum()
    if suspicious:
        print(f"{col}: {suspicious} sentinel nulls")

3. Cardinality#

python

for col in df.columns:
    print(f"{col:24s} {df[col].nunique():>8,} distinct")

What you are looking for:

1 distinct value — a constant. Useless, or a filter someone already applied and did not tell you about.
n distinct = n rows — an identifier, or a free-text field.
2–50 — a categorical. List every value. This is where you find "US", "us", "USA" and "United States" living together, as they do in dirty-customers.
Suspiciously round numbers — exactly 1,000 distinct customers often means the extract was truncated.

4. Distributions, per numeric column#

python

df.describe(percentiles=[.01, .25, .5, .75, .99]).T

Read the table for:

min below zero on something that cannot be negative — age, price, count.
max implausibly large — an age of 999, a duration of 40 days.
mean far from median — skew. Report medians.
an enormous p99 versus p75 — a heavy tail, which means averages will mislead and your charts need a log scale or clipping (clearly labelled).
zeros — are they real zeros or missing values wearing a costume?

5. Time coverage#

python

df["ordered_at"].agg(["min", "max"])
df.set_index("ordered_at").resample("D").size().describe()
df.set_index("ordered_at").resample("D").size().pipe(lambda s: s[s == 0])

Three specific things: does the range match what you were told; are there days with zero rows (an outage, or genuinely no activity — these look identical and are not); and does volume shift abruptly partway through (a migration, a tracking change, a backfill).

The first and last day are almost always partial. Exclude them from any rate calculation or your trend starts and ends with a cliff.

6. Cross-field consistency#

The checks that find real corruption, because they encode facts about the world.

python

(df["ordered_at"] > df["shipped_at"]).sum()                 # time travel
(df["units"] * df["unit_price_usd"] < df["revenue_usd"]).sum()  # arithmetic disagrees
(df["salary_min_usd"] > df["salary_max_usd"]).sum()         # inverted range
df[df["churned_at"] < df["signed_up_on"]].shape[0]          # churned before signing up

Every dataset has three or four relationships that must hold. Write them down and check them. They fail more often than you would like.

7. Duplicates, including near-ones#

Exact duplicates are easy. Near-duplicates — differing only by whitespace, casing, or a trailing character — are where the real count errors live.

python

exact = df.duplicated().sum()

normalised = df.assign(
    name_key=df["full_name"].str.strip().str.lower().str.replace(r"\s+", " ", regex=True),
    email_key=df["email"].str.strip().str.lower(),
)
near = normalised.duplicated(subset=["name_key", "email_key"]).sum()
print(f"{exact} exact, {near} after normalising")

dirty-customers contains 43 near-duplicates that differ only in casing and whitespace. Exact matching finds none of them.

8. Write the memo#

Ten lines, at the top of the notebook, before any analysis:

text

retail-orders, extracted 2026-08-01
  grain      : one row per order line (order_id + line_no unique: yes)
  period     : 2023-01-01 to 2024-12-30, no gaps > 1 day
  rows       : 7,000
  missing    : none material
  caveats    : revenue_usd is gross of returns; `returned` flags them separately.
               First and last day partial — excluded from daily rates.
  unknowns   : no timezone documented; assuming UTC.

That memo is what makes the analysis reviewable, and it is what you will thank yourself for in four months when someone asks where a number came from.