One score is 11 writes, and our cost estimate lost its working

Originally published on Hearso Writing on 2026-08-11. Hearso is the team trivia game I build and run; these posts are the engineering notes from building it.

One accepted score writes 11 items, not 1. Our $0.68-a-month estimate kept the answer, lost the unit price, and has never met a real bill.

One score submission does not cause one write. In our leaderboard's merged code, one accepted score writes 11 items and makes 3 strongly consistent reads.

The leaderboard is a small AWS service behind Hearso's trivia games: API Gateway, a few Lambda functions and one DynamoDB table. Rank is baked into the key each row is written with, and that key and its index are fixed from the first row onwards. So I wanted a defensible price per score while the table shape was still cheap to throw away.

Which is an awkward thing to admit about a cost estimate: the one we wrote down on 2026-08-11 kept the answer and lost the working.

So the question was plain. What does one score cost us to store, and how much of that number do we actually know?

I counted the writes by reading the merged handler, rebuilt the recorded arithmetic from its own inputs, and set both against the one real measurement we have: 1,000 submissions to one board on DEV, on 2026-08-19.

Short answer: the plan said 18 item writes and about $0.68 a month at 1,000 submissions a day. The code writes 11, spread over more places. The only measurement, on one board rather than three, used about 6.9 write units per submission where the plan predicted 2. And the $1.25-per-million unit price behind $0.68 appears nowhere in our repository; I had to work it out backwards.

There's also a question near the end, about why submissions failed under load. My first answer to it was wrong, and one corner of it is still open.

Why one score is not one write

The table design has its own post: We store every score backwards so DynamoDB never has to sort. Here I only need three ideas from it.

Write fan-out. Think of filing one form at an office that keeps a copy in every department it concerns, plus an index card for each copy. You filled in one form; the office did several filings. Write fan-out is that multiplier: how many stored items one logical action touches.

The billing unit. DynamoDB bills those filings in write request units. One unit covers writing one item of up to 1 KB; an item twice that size costs two. Our table runs on-demand, so there is no reserved capacity to plan and idle hours cost nothing. That suits a service that bursts when a game ends and then sits quiet.

The index is a second cabinet. The ranked order lives in a global secondary index, and DynamoDB maintains it with its own write every time the base row changes. Every index is another write, billed separately, for as long as the index exists. That is the biggest single reason our fan-out is what it is. Figure 1 shows the whole of it.

Figure 1. The write fan-out of one new-or-improving submission with the default board selection, counted by reading the merged handler and command-builder source at the merge commit of 2026-09-05.

The $0.68 estimate, rebuilt step by step

The original decision record priced the design at roughly $0.68 a month at 1,000 submissions a day. It recorded the result and not the working. A monthly write price is three things multiplied together: how many submissions arrive, how many billed units each one causes, and what a unit costs. The record kept the first two and the answer, and lost the third. Here is the working, reconstructed from the record's own inputs:

1. 1,000 submissions/day × 30 days = 30,000 submissions/month.
2. Nine conditional writes per submission (three periods × three variants) = 270,000 item writes.
3. Each one mirrored into the index = 270,000 more, so 540,000 write request units.
4. 540,000 ÷ 1,000,000 × $1.25 = $0.675, which rounds to the $0.68 in the record.

Every assumption that arithmetic rests on:

That last row is the one to be careful with. No price list and no date appears in our documents, so I worked backwards: $1.25 per million is the only rate that reproduces $0.68 from the record's own inputs. Treat it as my reconstruction, not a quoted price, and check current AWS pricing before reusing it. The table design post prices its own sum at $0.6250 per million, a rate it read from a worked example on AWS's pricing page. The two figures answer different questions: one is the rate the record's sum implies, the other a rate AWS printed.

The code grew in a different direction from the plan

Three things moved between the estimate and the merged code, and they moved in opposite directions.

The variant axis never grew. Only one variant exists, because the game has not shipped topics or difficulty levels yet. A fourth period, the calendar month, arrived later but stays opt-in.

Meanwhile each board got more expensive. A per-board counter row now holds a score distribution. So an accepted score commits the score row and the counter delta together in one transaction, after a strongly consistent read to learn what it is changing.

Figure 2 puts the two totals side by side. It counts item writes only. It leaves out the read, and the transaction's third item: a condition check against an admin lease, which stores nothing.

Figure 2. Item writes per accepted new-or-improving submission, counted from the decision record on the left and from the merged source on the right. Neither bar is a measured capacity figure.

The headline number never moved while all this happened. That is the part I would warn anyone about: an estimate ages against the code, quietly.

The only real measurement says 6.9 units, not 2

Here's the one number anyone has measured. On 2026-08-19 at 01:35:30Z we drove 1,000 new-player submissions at a single all-time board on DEV, at four levels of concurrency. At eight in flight all 1,000 succeeded. The table consumed 6,868 write capacity units in the peak minute, with zero throttled requests reported.

If that minute's consumption is all this run, that is roughly 6.9 units per single-board submission. The artifact does not prove it, since nothing recorded the table as otherwise idle. The plan's model predicts two: one score row, one index mirror.

A paper check points the same way. Applying AWS's documented rule that a transactional write costs two units per item, one board's path is: reservation 1, score row 2, counter row 2, index entry 1, outcome 1. That is 7 units, before the condition check. It is arithmetic, not a measurement, but it says the gap is the machinery added after the estimate, not noise.

That is why I would not quote $0.68 today without re-deriving it.

Why did submissions fail above eight in flight?

The same run says something sharper about concurrency, and it is the part of this story I got wrong first.

A hot partition is the supermarket with one till open. DynamoDB spreads work by key, so when every write lands on one key, that key's throughput is the ceiling, however idle the rest of the table is. All 128 of a board's distribution counters live on a single item, so every accepted score on a board touches the same key. Figure 3 shows what happened as we raised the pressure.

Figure 3. Outcome of 1,000 new-player submissions to one all-time board at four concurrency levels, DEV us-east-1, 2026-08-19T01:35:30Z, deployed commit e3fb772. The artifact records zero throttled requests for the 1,000-way and 8-way runs; its written decision states the 16-way and 64-way failures also happened without throttles. No failure reason was recorded per request; the run's CloudWatch metrics, read on 2026-09-28, show the failures above 8 in flight were mostly Lambda throttles at a concurrency limit of 10.

At 64 in flight, 475 of 1,000 submissions failed, and by the run's own decision note DynamoDB throttled nothing. The run read that combination as transaction conflict on the one counter row, not capacity. The rule it set: review sharding before more than eight simultaneous new or improving writes land on one board, or when the counter row reaches a quarter of the table's write consumption.

Update, 2026-09-28: it was the Lambda limit, not the counter row. The run recorded no failure reason per request, but CloudWatch keeps five-minute metrics for 63 days, longer than the logs. In them, Lambda concurrency sat pinned at exactly 10 while Lambda throttled 2,471 invocations. The API returned 2,562 server errors in the whole run, so at most 91 had any other cause. DynamoDB gives the same 91 on its own: 2,806 idempotency reservations, 2,715 committed transactions.

Conflicts were real, 1,177 of them, but the handler's retries absorbed all but those 91 at most. The one gap is the 16 level. It shared a five-minute bucket with the 64 level, so its 14 failures are consistent with throttles but can't be attributed request by request: that bucket's 65 handler failures could cover all 14.

We re-ran the same ladder on 2026-09-28 with a reason recorded on every request, against today's code (eight attempts per conflict, not three) and today's limits. At 16 in flight, conflict failed 1 request in 1,000. At 64 it failed 50, beside 216 Lambda throttles at the function's new cap of 50 and 428 refusals from the API key's usage plan. Conflict is the next limit, not the one we hit in August.

I got this wrong in the first version of this post, where I wrote the conflict reading as a finding. Here is how the two explanations came out:

Both stories fit the same four numbers. What told them apart was a metric nobody had opened: Lambda's own throttle count.

The wrong reading also left something behind. The sharding rule above was written for failures that the counter row mostly did not cause.

Where the evidence stops

So what would have to be true for the $0.68 to stand today? The fan-out would have to match the plan, and the units per write would have to match the model. The sections above say the first does not, and the one measurement says the second probably does not. The check that could settle it has not been made: nobody has measured the default three-board fan-out with per-request consumed capacity. The DEV run deliberately wrote to one board so the counter row was the only variable. The follow-up measurement was specified in some detail and never run: no commits, no report directory, nothing.

So, honestly: the fan-out count comes from reading code. The monthly price comes from an estimate whose unit price I had to reverse-engineer. The only capacity measurement covers a third of the boards a real submission writes.

Our written trigger for revisiting the fan-out is "write cost exceeds about $10 a month", against a $15 ceiling. I can say where we sit against that from a model, not from a bill.

Predictions vs outcome

Key takeaways

• Count writes from the code, not the plan. Our plan said 18 and the code writes 11, in places the plan never had.
• Price every index as a second table. Each one is another write on every change to the base row, for as long as it exists.
• Write down the working, not just the answer. Our record kept $0.68 and lost the unit price, so it cannot be updated, only re-derived.
• Re-check an estimate whenever the write path changes. Ours aged in both directions while the headline number stayed put.
• Record why each request failed. Zero throttles is an absence; without a reason per request, "conflict" and "a concurrency limit" look the same. Ours was the concurrency limit, sitting in a metric we never opened.

What I'd do next

1. Measure a real default submission end to end: ReturnConsumedCapacity on every command in the path, logged, totalled for one three-board submission.
2. Take load off the two board-wide rows every submission touches, the seed-lock check and the histogram row. With the Lambda limit raised, conflict on them is the next limit: at 64 in flight it failed 50 of 1,000 even after eight attempts.
3. Repeat that sweep against the three-board fan-out: three counter rows per submission is three chances to conflict, not one.
4. Put a dated price and a link in the cost estimate, so the next person updates it instead of re-deriving it.
5. Check the estimate against a real bill before the fan-out widens, because the variant axis is one configuration change away from tripling.

Evidence

All times UTC.

• Nine writes per submission, mirrored into the index, "~$0.68/month at 1,000 submissions/day": docs/adr/0009-inverted-score-key-design.md, Consequences. The record states the result only; steps 1 to 4 above are my reconstruction, and the $1.25-per-million unit price appears nowhere in the repository.
• On-demand billing and a single index: docs/adr/0010-table-and-index-design.md: BillingMode: PAY_PER_REQUEST, RankIndex projection INCLUDE (displayName, country, lastPlayedAt).
• Three periods, one variant, in the merged code: src/lib/score-update-command-builder.js (DEFAULT_BOARD_PERIOD_KINDS = alltime, daily, weekly; BOARD_PERIOD_KINDS adds monthly as opt-in) and src/config/leaderboard-fanout-config.js (DEFAULT_LEADERBOARD_VARIANTS = ['alltopics'], widened only by deployment configuration).
• The 11 item writes: src/handlers/submit-score-handler.js: one reservation PutCommand, one GetCommand per board with ConsistentRead: true, one TransactWriteCommand per board, one outcome UpdateCommand. The transaction's three items come from buildAtomicScoreMetadataWrite in src/lib/leaderboard-metadata-counter.js: a lease ConditionCheck storing nothing, the score update, and the counter delta.
• 6,868 write capacity units in the peak minute, 1,000 of 1,000 succeeded at concurrency 8, 0 throttled requests: benchmarks/LFL26LEADBORD-035/metadata-counter-dev.json, measured 2026-08-19T01:35:30.415Z, deployed commit e3fb772, operation "new-player POST /scores with boards=[alltime]".
• The 7-unit paper check: derived, not measured. The two-units-per-item rule for transactional writes is from the DynamoDB Developer Guide, "DynamoDB Transactions: how it works", as quoted in the table-design post; the item list is the handler source above.
• 204 / 525 / 986 / 1,000 successes at 1,000 / 64 / 16 / 8 in flight, and the sharding trigger: same file, results and decision; narrative in benchmarks/LFL26LEADBORD-035/README.md. Reviewed as leaderboard PR #52, whose own summary records the 1,000-way burst committing 204 of 1,000 with zero throttles.
• The Lambda concurrency limit: the overnight handoff for 2026-09-26, entry 2026-09-27 ~10:55Z (aws lambda get-account-settings: ConcurrentExecutions = 10), and entry ~14:20Z (the quota increase approved, account concurrency 1,000, request closed 12:16Z). The 2026-08-19 run's own metrics show concurrency pinned at 10 then too (next item).
• Lambda throttles, not conflict, on 2026-08-19, and the re-run on 2026-09-28: leaderboard PR #102, benchmarks/ice-breaker-147-submission-sweep/README.md and its cloudwatch.json. The 2026-08-19 run's five-minute CloudWatch metrics: peak ConcurrentExecutions 10, 2,471 Lambda throttles, 2,562 API 5xx, 2,806 reservations against 2,715 committed transactions, 1,177 TransactionConflict. The re-run repeats the ladder with a reason recorded per request.
• The measurement that never ran: docs/archive/specifications/ticket-054-logging-specification.md specifies per-request consumed-capacity logging for LFL26LEADBORD-054; git log --all returns no commit mentioning it and reports/ contains no directory for it.
• The $10 trigger and the $15 ceiling: docs/PRD.md, deferred-items table ("Reducing 9-writes-per-submission amplification: write cost exceeds ~$10/month") and operating constraints.