Field note · October 24, 2023
Yesterday keeps changing and that is correct
Late-arriving events, a seven-day lookback, and how to tell finance about it without a crisis.
We moved a revenue table from processing time to event time, which was the right change, and immediately created a political problem: yesterday's number now moves for a couple of days afterwards.
Finance noticed within a week and were, reasonably, alarmed. A number that changes after publication looks like a bug.
It is not a bug. Events arrive late — a phone was offline, a partner batched their export, a vendor's job failed and reran. Attributing them to the day they happened is correct, and correct attribution means the past is revised slightly as reality catches up.
Two things fixed it, and only one of them was technical.
The technical part is a lookback window. Every night we reprocess the last seven days rather than one. Idempotency makes this free:
delete from daily_revenue where order_date >= current_date - 7;
insert into daily_revenue
select order_date, sum(revenue_usd) from orders
where order_date >= current_date - 7
group by order_date;We picked seven by measuring: the distribution of _loaded_at − ordered_at covered 99.4% of events within six days.
The organisational part was the sentence "figures are provisional for seven days and firm after that", added to every report footer and said out loud in one meeting.
That sentence ended the problem entirely. The alarm was never about the number moving. It was about being surprised.