A working setup that will not embarrass you

You can do a lot of good work in a notebook and a laptop. You cannot do reliable work that way, and the difference shows up at exactly the wrong moment: when someone asks how a number was produced, eight months later, and the notebook has been re-run in a different order since.

This lesson is the minimum setup. It is deliberately boring.

Version control, including for analysis#

Everything goes in git. Queries, notebooks, dashboard definitions, the one-off script. Not because you need branching strategy for a scratch analysis, but because git is the only mechanism that reliably answers "what did this look like when we published that number".

bash

git init
git add . && git commit -m "Retention analysis, first cut"

Notebooks are awkward in git because the JSON diff is unreadable. Two workable answers: strip outputs before committing (nbstripout), or keep the logic in .py modules and let the notebook be a thin presentation layer that imports them. The second is better and almost nobody does it at first.

One environment per project, declared in a file#

The failure mode here is not exotic. It is: your analysis works, you hand it over, it does not work, and three hours vanish into a pandas version difference.

bash

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install pandas duckdb matplotlib
pip freeze > requirements.txt

uv or conda are fine substitutes. The requirement is not a specific tool, it is that the environment is written down in the repository rather than living in your shell history.

Notebooks: what they are good and bad at#

Notebooks are excellent for exploration and terrible for anything that must run twice. The specific danger is hidden state: cells run out of order leave variables defined that the code no longer produces, so the notebook works on your machine and nowhere else.

Rules that make them safe:

Before sharing or trusting a result, restart and run all. If it does not survive that, it is not a result.
No cell should depend on a cell below it.
The moment a function is used twice, it moves to a .py file.
Notebooks are for finding the answer. A script or a model is for producing it.

Read data once, cache it, and never hit the warehouse in a loop#

python

from pathlib import Path
import pandas as pd

CACHE = Path("data/retail-orders.parquet")

def load_orders() -> pd.DataFrame:
    """Fetch once, then read from local parquet on every rerun."""
    if not CACHE.exists():
        CACHE.parent.mkdir(parents=True, exist_ok=True)
        df = pd.read_csv("https://example.com/retail-orders.csv", parse_dates=["ordered_at"])
        df.to_parquet(CACHE)
    return pd.read_parquet(CACHE)

Parquet over CSV for anything you will read more than once: it is columnar, typed, and roughly five to ten times smaller. CSV is a transport format, not a storage format — it forgets every type it ever had.

DuckDB is the answer to "I need SQL but not a warehouse"#

For anything up to tens of millions of rows on a laptop, DuckDB removes the whole question of infrastructure.

python

import duckdb

duckdb.sql("""
    select category,
           count(*)                as lines,
           round(sum(revenue_usd)) as revenue
    from 'data/retail-orders.parquet'
    group by category
    order by revenue desc
""").show()

It queries CSV and Parquet files directly, in place, with no import step. For learning SQL — including everything in module four of this course — it is the shortest path from nothing to a working query engine.

A project layout worth copying#

text

project/
  README.md          <- what this is, how to run it, who to ask
  requirements.txt
  data/              <- gitignored; raw inputs, never edited by hand
  sql/               <- queries as files, not pasted into notebooks
  src/               <- reusable functions
  notebooks/         <- exploration, numbered by order of creation
  output/            <- generated charts and tables, also gitignored

The two rules that matter: raw data is never edited in place, and generated output is never committed. Both are recoverable from code; if they are not, the code is incomplete.