We rewrote our interview loop and the hires got better

Our old loop: a SQL screen, a take-home, a stats interview, a culture chat. Reasonable-looking. It kept producing people who were technically fine and struggled with the job.

The pattern in the misses was consistent. They could write the query. They could not tell that the question they had been given was the wrong question.

What we changed:

Replaced whiteboard SQL with one question containing a grain trap, done together with the schema visible and search allowed. The signal is not whether they write it — it is whether they ask "what does one row of this table represent?" before writing anything. That question is the strongest predictor in the whole loop and it takes ninety seconds to observe.

Added a scoping stage. A deliberately vague request: "leadership wants to know if enterprise customers are healthy." We assess what they ask before touching data. This is the highest-leverage skill in the job and our old loop tested it zero times.

Added a debugging scenario. "Conversion dropped 18% on Tuesday, walk me through what you check." Good candidates check the pipeline before the analysis. Great ones ask whether the metric definition changed.

Cut the take-home to two hours and actually enforced it. The old one was nominally four and realistically eight, which selected for free time.

Wrote down what each stage tests, before running it. This was the most boring change and possibly the most valuable — we discovered we were testing SQL three times and communication never.

Eighteen months and a small sample, so treat this as an anecdote rather than evidence. But the two people we hired under the new loop have both been unambiguous successes, and the thing they have in common is that they push back.