Our benchmark passed at 239 ms p95, and we threw it away

Originally published on Hearso Writing on 2026-08-18. Hearso is the team trivia game I build and run; these posts are the engineering notes from building it.

Our first leaderboard propagation run passed its 1 s target and still left 100 probe players on every board, so no replay could repeat it.

Our first leaderboard benchmark passed with a p95 of 239 ms and zero errors. I threw it away anyway, because nobody could run it again and get the same experiment.

The leaderboard is a small AWS service: API Gateway, a few Lambda functions, and a DynamoDB table whose secondary index holds the ranked order. I wanted one number from it. How long after a player submits a score can anyone actually see it on the board?

It's an odd feeling to reject a run that passed. Keeping it would have been easier, and one of our documents still does (more on that below).

So the question shifted from "is it fast enough?" to "is a passing run a baseline if the next run starts from different data?"

I ran the repository's own propagation runner twice on 2026-08-18, against the DEV stack in us-east-1 with response caching off, 300 samples each time.

Short answer: no. The first run left its 100 measurement players behind on every board, under a board name that never changed, so a replay would begin from 200 rows instead of 100. The fix was a run-scoped board name: a config change plus about 48 lines of runner code.

The accepted second run measured p50 159 ms, p95 210 ms and p99 322 ms against a 1-second target, with no sign of slowing from 100 to 10,000 players.

How do you time a score becoming visible?

Think of the main table as a filing cabinet sorted by player, and of a second cabinet a clerk keeps sorted by score. That second one is a global secondary index. DynamoDB refreshes it in the background rather than as part of your write.

So a write that already returned 200 and a score that is readable in rank order are two different moments. The gap between them is what this benchmark measures.

The tool is npm run benchmark:propagation. Each sample starts the clock immediately before POST /scores and stops it when that player appears in GET /leaderboards.

A p95 promise needs a p95 measurement

Line every sample up slowest-last. The p95 is the value 95 percent of them came in under; p50 is the middle sample and p99 is near the worst. In the accepted run p50 was 159 ms and p99 was 322 ms, about double. An average would have quietly hidden the part players notice.

A service level objective is a promise with both a number and a percentile in it. Ours: a submitted score is readable through the regional API within one second at p95.

We measured 210 ms, leaving 790 ms of headroom: the slack between the promise and the measurement. Headroom is what tells me whether the next feature on this path is affordable. Figure 1 shows the accepted run per board size.

Figure 1. Propagation percentiles per prepared board size, 100 samples each (300 total), DEV us-east-1, run started 2026-08-18T19:38:19Z. Source: benchmarks/20260818T193819.881Z/.

The run that passed and couldn't be repeated

The numbers above are from the second run. The first finished at 09:41Z with 300 samples, p95 239 ms and zero errors. It passed. Then I worked out how to repeat it, and realised I couldn't.

That matters because of what a baseline is for. It exists to have a later run compared against it.

A benchmark is a chemistry experiment, and the first rule is that the beaker starts clean. Isolation here means every replay begins from the same state.

Here's what the run actually did. It seeds a board to a given size, then submits 100 deterministic measurement players and times each one's appearance. Those 100 probes stay on the board afterwards. And the board's name was fixed: propagation-slo-v2-{leaderboardSize}, the same every time.

So the "100-player" board would greet the next replay with 200 rows. The measurements were true for that one moment and useless as a baseline, so I rejected the run. Figure 2 shows the flaw and the fix side by side.

Figure 2. The reproducibility flaw and its fix. Board names are the game-ID templates recorded in each run's config.json.

The fix: put the run's identity in the board's name

The corrected harness makes the namespace part of the experiment instead of part of the cleanup. A run-scoped namespace is a board name with the run's identity baked in, so two runs can never touch the same rows:

prop-v1-{runId}-{leaderboardSize}

{runId} becomes a base-36 encoding of the Unix time in milliseconds when the run starts. The committed config records the strategy, the generated ID and the template it resolved to: here prop-v1-msz2cjis-{leaderboardSize}.

An earlier version of this post called the fix "one line". It wasn't. Commit adc4b27 is a config change plus about 48 lines of runner code that resolve and record the run ID. Still small, but the runner had to learn a new idea, and I undersold that.

A cleanup instruction in a runbook is a wish. A namespace rule in the executable config is a fact.

What the accepted run says

Zero errors, and all 300 samples resolved on the first poll. Post-run verification found 200, 1,100 and 10,100 rows: each prepared size plus its 100 probes, exactly as designed.

Figure 3 sets the two runs side by side. Don't read the gap between them as an effect of the fix: the runs differ in board sizes, sample counts, seed score range and deployed commit.

Figure 3. Percentiles of the rejected 09:41Z run and the accepted 19:38Z run, 300 samples each, 2026-08-18. The two runs differ in board sizes, sample counts, seed score range and deployed commit, so the gap between them is not a measured effect of the namespace change.

I also would not read "bigger boards are faster" off the table. The sizes ran one after another, each distribution holds only 100 samples, and ordinary variance moves percentiles around. The defensible claim is narrower: going from 100 to 10,000 prepared players produced no evidence of a regression.

The slowest sample in the run was 1,533 ms, and it was sample 1 of the first board. The first end-to-end score this service ever recorded on real AWS took 3,265 ms (leaderboard PR #30). So a slow first sample is a pattern here, but a pattern is not a diagnosis, and I never established this one's cause.

The decision record still cites the run I rejected

The decision record that sets this SLO still quotes the rejected run: 239 ms, 761 ms of headroom, pointing at the 09:41Z folder. I checked again on 2026-09-28 and it still does.

Correcting it means writing a superseding record, and I haven't. The verdict is the same either way. But the citation sends a future reader to data I decided not to stand behind, which is exactly the kind of quiet drift this post is about. Rejecting the run fixed the measurement. It did not fix what already pointed at it.

Freshness at the edge is a different promise, still unmeasured

Freshness is a separate promise for a separate seam: six seconds at p95 through the CDN edge. That is the one-second propagation budget plus up to five seconds of cache age.

Caching is still off, so that number is not measured. Reporting a regional result as an edge result would fold two failure modes into one comfortable figure.

Predictions vs outcome

Key takeaways

• Put isolation in the config, not the runbook. If the system under test keeps data, a benchmark is only as trustworthy as its namespace.
• List every stateful input before calling a run a baseline. Population, namespace, cache path, target commit and the seam you measure at all count.
• Quote the percentile you promised, and the headroom left. Ours: p95 210 ms against a 1 s target, so 790 ms spare.
• Treat "passed" and "defensible" as two separate checks. Both our runs passed; only one can be replayed.
• Fix the citations when you fix the measurement. Our record still points at the run we threw away.

What I'd do next

1. Write the superseding decision record, so the SLO cites the accepted run instead of the rejected one.
2. Interleave the board-size order, so a size comparison is not confounded by run order.
3. Raise samples per size above 100 before quoting a p99 per size; at 100 samples, p99 rests on a single measurement.
4. Measure freshness at the edge once response caching is on, as its own run, never inferred from this one.

Evidence

All times UTC.

• Accepted run, 300 samples, p50 159 / p95 210 / p99 322 ms, 0 errors: benchmarks/20260818T193819.881Z/report.md; recomputed from the 300 lines of raw.jsonl, which also show ok: true and polls: 1 on every sample.
• Per-size percentiles (160/235/449, 158/181/206, 158/210/265 ms): same report and raw file; each size holds exactly 100 samples.
• Slowest sample 1,533 ms: raw.jsonl, first line, {"durationMs":1533,"leaderboardSize":100,"sample":1}.
• Run configuration and isolation: benchmarks/20260818T193819.881Z/config.json: gameIdTemplate prop-v1-{runId}-{leaderboardSize}, runIdStrategy unix-millis-base36, resolvedGameIdTemplate prop-v1-msz2cjis-{leaderboardSize}, commit 7e8a1cd, propagationTargetP95Ms 1000. Resolution logic: tools/benchmark-runner/run-propagation.mjs.
• The size of the fix: commit adc4b27, 2026-08-18, "LFL26LEADBORD-033: commit the first benchmark baseline": the config change plus about 48 lines of runner code (claims audit, 2026-09-28).
• Post-run populations 200 / 1,100 / 10,100: reproducible configuration section of the accepted run's report.md.
• Rejected run, p50 162 / p95 239 / p99 338 ms: benchmarks/20260818T094113.895Z/report.md and its 300-line raw.jsonl; its config.json carries gameIdTemplate propagation-slo-v2-{leaderboardSize} with no run ID, sizes 100 and 1,000, 150 samples each, seed scores 1 to 1,000, commit 6fe0834.
• The SLO, its 1-second target, and the stale citation: docs/adr/0012-propagation-and-freshness-slos.md (LFL26LEADBORD-031), which records 239 ms and "761 ms of observed headroom" and points at benchmarks/20260818T094113.895Z/. Re-read on 2026-09-28 in a local checkout of the develop branch; unchanged.
• Freshness is gated, not measured: SLO verdict table in the accepted run's report.md: "GATED. Caching is disabled".
• The two runs as they were reviewed: leaderboard PR #47 (the rejected run's measurement) and PR #49 (the baseline, recording the probe-row skew and the namespace fix). The 3,265 ms first live write is PR #30.