- Cache Warmth Masquerading as a Fix

A performance report claimed a double-digit percent speedup from a batch of code changes. Upon looking more carefully proved the changes had nothing to do with it.

Re-running the same workload twice in a row will usually look faster the second time, because the database's buffer cache is now warm — and that effect alone can produce numbers that look exactly like a real fix.

The tell was in the query count: it was identical before and after the supposed optimization. If the code had actually changed how data was fetched, the number of queries or rows touched should have moved too. It didn't. What had moved was which run went first. The "before" run was cold; the "after" run inherited a warm cache from the run just before it — and when the same comparison was repeated with the run order flipped, the direction of the "improvement" flipped with it.

Digging further showed the effect wasn't small: across several workload types, cold-versus-warm run times differed by anywhere from under 2x to well over 7x, depending on how much of the workload was memory-bound. Any single before/after number, taken without knowing whether it was measured cold or warm, was close to meaningless.

Fixing this meant changing the measurement process, not the code:

• Starting to compare cold-to-cold or warm-to-warm, never cold-to-warm.
• Running each configuration multiple times and reporting a range instead of single number.
• Looking for another independent signal (query count, rows processed) that should move if the "fix" is real — a speedup with no change in underlying work is a big red flag.

A run that finishes in 13 minutes vs one that finishes in 16 minutes; look like a 20% regression — until you learn the second run happened right after a service restart which flushed every cache, and the first didn't.

Note to self: a performance win that isn't accompanied by a matching drop in actual work done (queries, rows, bytes) is a caching artifact until proven otherwise — verify before you ship the "fix," and definitely before you tell anyone about it.