Field note · April 25, 2023
The notebook that only ran once
Hidden state, a board number, and why "restart and run all" became a rule.
Someone asked me to reproduce a churn figure from four months earlier. I had the notebook. I ran it. I got a different number.
Not slightly different. 14% versus 19%.
The cause took a day to find: a cell defining a filter had been edited after being run, and never re-run. The variable in memory when the original number was produced no longer matched any code in the file. The notebook had never, in its entire life, run top to bottom successfully.
It had been in a board deck.
Three rules since, and they are not negotiable on my team:
Restart and run all before you trust anything. If it does not survive that, it is not a result. This catches the whole class of problem in about ninety seconds.
No cell may depend on a cell below it. Notebooks are a linear document pretending to be a program; keeping them actually linear removes most of the danger.
The moment a function is used twice, it moves to a .py file. Then it can be tested, imported, and diffed in a way that a JSON blob cannot.
None of this makes notebooks bad. They are the right tool for exploration and I use them daily. They are simply the wrong tool for producing a number someone will act on, and the boundary between those two activities is invisible while you are crossing it.