Originally published on Hearso Writing on 2026-09-18. Hearso is the team trivia game I build and run; these posts are the engineering notes from building it.
Production games still sync by polling every 1.5 s. How an answer reaches every screen, the WebSocket path built to replace it, and where the points land.
A player taps an answer, and four other screens have to agree on what just happened. In Hearso's production games they find out by asking: every open page polls the server every 1.5 seconds.
Hearso is standup icebreaker trivia. A host opens a room, teammates join with a short code, and ten ten-second questions run on a server-authoritative clock. Scores from a finished game land on a shared leaderboard in its own AWS account.
On 2026-09-18 two coding agents built the poll's replacement overnight: a Rust WebSocket service called hearso-rt, run on one laptop. I'll be upfront that it still isn't what players use. Ten days later, every new production room is still polled.
So I wanted to write both paths down end to end, with nothing hand-waved: what happens between the tap and the other screens, what a socket changes, and where the points go afterwards.
The sources are the three repositories themselves, read on 2026-09-18, and the shared log the two agents kept while they built the new path (the cutover log).
Short answer: the poll shows a change up to 1.5 s late (750 ms on average) and costs a four-player room 160 requests a minute just to stay current. The socket path cuts that to 24 pings and pushes the exact JSON the poll returns, so the page's rendering code didn't change.
The points take their own road: server-only, to a DynamoDB table sorted by its own key, where a benchmarked score was readable 159 ms after submission at the median.
Where things stand on 2026-09-28
This post was written the night the socket path was built, when everything here about the socket was true of a laptop and everything about polling was true of hearso.com. Several things have moved since. Here is the status in one place; the sections below describe the design as it was read on 2026-09-18.
| On 2026-09-18 | On 2026-09-28 | |
|---|---|---|
Where hearso-rt runs | One laptop, reached over loopback only | A VPS |
| New production rooms | Polled | Still polled |
Scores from a ws game | Reach no leaderboard | A separate relay forwards them; not yet proven with a real production game |
| The leaderboard's production stack | Doesn't exist | Exists |
| The two-hour soak | Unfinished at minute 90 | Passed all 14 conditions at 21:53:43Z that night |
| Is the service ready? | /health only | Real readiness, plus a heartbeat it writes to the database (deployed 2026-09-28) |
Update, 2026-09-28: the rows above replace the separate update notes this post used to carry. The ticket-order and one-game-id items near the end were not re-checked for this update.
Seven words this post leans on
| Word | What it means here |
|---|---|
| room | One shared space, opened by a host. It moves lobby → question → reveal → finished exactly once. |
| host | The player who opened the room. Starts it; the game advances itself from there. |
| round | One question. Ten rounds make a game. |
| snapshot | The whole room as one player should see it now: the same JSON the poll returns. |
| ticket | A 30-second, single-use credential that lets one browser open one socket. |
| outbox | An ordered table of finished games, written inside the finish transaction. |
| transport | Which service drives a live room: poll (Next.js) or ws (hearso-rt). |
The switch between the two transports is a per-room column, rooms.transport. It's stamped once at room creation, and it's poll unless a deployment explicitly says otherwise.
Figure 1 is the whole system on one page; the rest of the post walks its arrows in order.
Figure 1. The pieces, read from the source on 2026-09-18. Solid lines ran in production; dashed lines ran on one laptop that night. The signed callback tells Next.js to drain the outbox and award badges.
The poll is still how production works, and it's a good first version
Here's the path of one tap. The browser sends POST /api/rooms/[code]/answer with a choice index. The route reads the room, refuses if no question is open, then makes one conditional write.
That write adds this round's answer to the player's answers object and raises their score, but only WHERE NOT (answers @> '{"1": {}}'): only while this round's key is still absent. Two taps race, one wins, and the other gets the winner's receipt.
Timing is never the browser's business. The room row carries question_started_at, stamped server-side, and the route computes the elapsed milliseconds itself. A correct answer is worth 100 + round(100 × (10,000 − elapsed) / 10,000), so 900 ms earns 191 and a 9-second answer earns 110.
Everyone else finds out on their next poll. GET /api/rooms/[code]/state runs every 1.5 seconds from every open page, and it isn't only a read: it's the game loop.
A poll that finds a question older than its 10-second limit plus a 2-second grace flips the room to reveal. One that finds a stale reveal advances it, or finishes the game after the last round. Those are conditional updates too, so polls arriving at once collapse into one transition.
I'd build it again. There's no connection lifecycle, no second process to watch, and nothing that can wedge a room, because there's no single driver to lose.
Why bother with a socket? Stale reveals, and traffic that scales with people
Two costs, and only one is latency.
The first is staleness you can't design away. A change lands in Postgres at some instant, and every other screen learns about it on its next poll: up to 1,500 ms later, 750 ms on average. That's fine for a roster. It isn't fine for a reveal, which is the moment the whole room is supposed to share.
The second is traffic that scales with people rather than events. At a 1.5-second interval each player makes 40 requests a minute (60,000 / 1,500), whatever is happening. On a socket the keep-alive is one ping every 10 seconds: 6 a minute per player.
Figure 2. Arithmetic, not a measurement: 60,000 / 1,500 = 40 and 60,000 / 10,000 = 6 per player per minute, from `POLL_MS` in the room page and `heartbeatMs` in the welcome frame. It counts the keep-current traffic only.
Both numbers in Figure 2 are exact derivations from two constants in the source, not a benchmark. In production there was no socket to benchmark.
The ticket: a cloakroom token for a service that can't read your cookie
A socket to a second service raises a question the poll never had to answer: how does that service know who is connecting?
Think of a cloakroom. You hand over your coat and get a numbered token; the counter never needs your name. The token is the whole claim, worth nothing to anybody else, and spent the moment it's used.
Precisely: a ticket is a short-lived bearer credential signed by one service and verified by another with no callback. Ours is v1.<payload>.<signature>, the signature HMAC-SHA256 over "v1." + payload. The payload is compact JSON with its keys in a frozen order: {"v":1,"jti":…,"sub":…,"room":…,"aud":"hearso-rt","iat":…,"exp":…}.
It lives 30 seconds (exp = iat + 30000), the verifier forgives 5 seconds of clock skew, and the first use burns its jti.
We needed one because the room actor cookie is httpOnly and path-scoped to /api/rooms/<code> on our own origin, so a handshake to another origin carries none of it. Next.js can read that cookie; the Rust service can't.
So the browser asks Next.js, which checks the cookie, that the room is ws, and that the seat is still held (left_at IS NULL). The browser then presents the ticket in the socket's address.
Figure 3. One connection attempt, read from the source on 2026-09-18. `K7M2Q9XP` is the room code in our golden fixtures, not a real room.
The burn in Figure 3 is one Valkey command: SET ticket:<jti> 1 NX PX 40001. NX only succeeds if the key is absent, so a second use gets nil and is refused with AUTH_TICKET_REPLAYED.
Both sides pin the same frozen test vectors. A change to either implementation that would break the other fails a test rather than a game.
One round over the socket has exactly one writer
This is the trade the poll never made. The poll has no single driver to lose, and a ws room has exactly one writer. Each room gets an owner task: a tokio task with a bounded mailbox (capacity 32) that every command and every timer for that room passes through, one at a time. There's no cross-task locking, because there's nothing to lock against.
The answer command is {"v":1,"id":"<uuid>","type":"room.answer","data":{"index":1,"choice":1}}, at most 4,096 bytes. index is there for one reason: a frame can arrive after the room has moved on, and naming the round is what lets the service refuse a stale answer instead of scoring it against the next question.
Figure 4. One round of a four-player game, read from the source on 2026-09-18.
The database write, third in Figure 4, is the poll route's shape exactly: a SELECT … FOR UPDATE on the room at the expected index, then an UPDATE guarded on the round key still being absent. The owner task then rebuilds the room for every viewer and publishes it. That publication is the part I like most.
A snapshot travels over a tokio::sync::watch channel, which holds exactly one value: the latest. Think of a whiteboard rather than an inbox. The writer overwrites it, and a reader who looks away and comes back sees the current state, not a queue of what they missed.
A page busy rendering during three rapid changes receives one frame carrying the newest room; the two it missed are gone. That is safe only because a snapshot is the whole room and not a change to it, so a skipped frame holds nothing the next one lacks. The client also drops anything whose version isn't greater than what's on screen, so a room can never walk backwards. That's how memory stays bounded when a phone throttles a background tab.
Update, 2026-09-28: one thing this section left out. The owner task also reloaded its room from the database on a 250 ms timer. Beside the database that cost almost nothing; 38 ms away, on 2026-09-23, it pushed the median answer reply to 3,658 ms. The polling was removed in the service's PR #11, and on 2026-09-27 the median was 328 ms against the production database. That story has its own post.
The same JSON on both pipes, so the page didn't change
The rendering code didn't change, and that was the design. It's the strangler approach: grow the new path around the old one, behind a switch, until the old one can go.
room.snapshot carries the exact body GET /api/rooms/[code]/state returns for that viewer, with one key added: "transport":"ws". Commands mirror the seven POST routes, and a cmd.result carries the status and body that route would have returned.
So the page has one function that either does a fetch or sends a command, and every call site reads ok, status and json() as before. A reply that never comes is status 0, which is what a failed fetch already was.
The handover is one branch. The first poll of a ws room comes back saying "transport":"ws", the page opens a socket, and the interval stops. A polled room's payload has no such key.
A room never changes transport, and nothing updates the column after the insert. That's what lets both sides assume a single writer for a room's whole life. Every reader treats an absent column, a null and an unrecognised value as poll, because on a database the migration hasn't reached, nothing could have stamped a ws room.
A finished game is recorded exactly once, and announced in the same transaction
Everything so far stays inside the room. The end of a game is the one moment that has to leave it, for badges and for the leaderboard.
When a ws game ends, hearso-rt runs one Postgres transaction. It holds the conditional UPDATE … SET status='finished' that decides the race, a games ledger row, a player_profiles bump per player, one plays row per ranked player, and one row in game_finished_outbox.
An outbox is a table written inside the same transaction as the thing it describes, so the record and its announcement can't disagree. Ours holds {"v":1,"roomCode","gameRound","finishedAt","plays":[{"playerId","rank"}]}, keyed by an always-growing seq, unique on (room_code, game_round, finished_at).
Badges stay in Next.js, because the achievement rules live there. Writing them again in another language in another repository is how two writers come to disagree about what a badge means.
After the commit, the service calls POST /api/internal/games/finished with {"outboxSeq":N}. It's signed v1=<hex HMAC-SHA256(secret, "<unix ms>.<raw body>")> and refused outside 60 seconds either way.
The route drains every pending row up to that seq in ascending order: award first, then mark, the mark conditional on the row still being unclaimed. A row it can't handle stops the drain, and the answer reports an offset below it.
That's the difference between a committed offset and a queue that leaves a hole nobody notices, until somebody asks why one player's first game was never recorded.
Here's a real example. The night it was first wired end to end, the callback drained two finishes in 0.12 s and 0.06 s. The same call swept up two older rows, 741.85 s and 915.10 s after their finishes (cutover log, web-side entry, 2026-09-18T11:57:48Z).
Where the points go: server-only, and sorted by the key itself
The callback above carries badges. Points take a different road, and on 2026-09-18 only a poll game's points travelled it.
Only a server submits a score. The leaderboard service has no player authentication by design: its trust model is that the calling game vouches for its players. So the deployment boundary is the trust boundary, and lib/leaderboard-client.ts throws at module load if it finds itself in a browser.
A finished poll game sends one POST /v1/scores per player with an Idempotency-Key of trivia-<roomCode>-<playerId>, so a retry is safe. API Gateway hands it to one Lambda: nodejs22.x on arm64, 256 MB, a 10-second timeout.
That Lambda writes three boards by default: all-time, the achievedAt day and its ISO week. They're computed from when the score was achieved rather than when it arrived, so a score set at 23:59:58 lands on that day's board. We ask for the day's board only when the player is a guest, to keep write costs down.
Each board is a DynamoDB item keyed playerId (partition) and leaderboardId (sort), where leaderboardId is <gameId>#<variant>#<period>: say trivia#alltopics#daily-2026-09-18. The write is conditional on attribute_not_exists(#score) OR #score < :score, so a submission that beats nothing moves lastPlayedAt and nothing else.
Both tables bill per request. A TTL on expiresAt retires daily rows 35 days past their period end, and weekly and monthly rows after about 400. Nothing streams off the table: there's no consumer downstream of the write.
The interesting key is the other one. Imagine a library where, instead of sorting the shelf when somebody asks, you choose each book's spine label so that shelving it alphabetically is the ranking.
That's rankKey: <invertedScore>#<achievedAtMillis>#<playerId>, each number zero-padded to 13 digits, the score stored as 1,000,000,000,000 − score. A score of 191 achieved at epoch millisecond 1789700000000 becomes 0999999999809#1789700000000#<playerId>.
A higher score gives a smaller string, and so does an earlier timestamp. So one ascending read of the RankIndex global secondary index answers "highest first, earliest wins ties", with no comparison logic downstream.
Figure 5. Propagation from `POST /v1/scores` to the score being readable: p50 159 ms, p95 210 ms, p99 322 ms over 300 samples, DEV stack in `us-east-1`, run started 2026-08-18T19:38:19Z, error rate 0.00%. The three board sizes ran one after another, so this is not a controlled comparison between them (online-leaderboard PR #49).
| Board size | p50 | p95 | p99 |
|---|---|---|---|
| 100 players | 160 ms | 235 ms | 449 ms |
| 1,000 players | 158 ms | 181 ms | 206 ms |
| 10,000 players | 158 ms | 210 ms | 265 ms |
| All 300 samples | 159 ms | 210 ms | 322 ms |
Worth setting against that: the very first score we ever wrote to real AWS took 3,265 ms on a cold-start sample (online-leaderboard PR #30), about twenty times the warm median. A benchmark measures a warm system, and the first request of the day isn't one.
What wasn't done on 2026-09-18, and what has moved since
The quickest way to judge a design like this is to ask where it would break. This is the list as written that night, with each item's status now. Where I haven't re-checked an item, the table says so.
| On 2026-09-18 | Status on 2026-09-28 |
|---|---|
Nothing was deployed. No VPS, no TLS, no DNS. The Rust service had only been reached over loopback, under a bot swarm on one laptop rather than traffic. The migration that adds rooms.transport hadn't reached a hosted database, and the web work sat on a local branch 45 commits ahead of develop. | hearso-rt runs on a VPS. New production rooms are still polled, so real players don't reach it yet. |
A ws game's scores never reached the leaderboard. The Rust service wrote plays, wrote the outbox row and called the callback, but had no score submission at all, and submitted stayed false on every seat of a ws room. | A separate relay reads the finish outbox and forwards ws scores to the leaderboard. It hasn't yet been proven with a real production game. |
The chaos harness was a stub. crates/sim was 43 lines; none of the eight invariants the plan asks it to check existed. The service's own scaffold pull request said so from the start: its green sim result "only proves the empty crate builds". | Later that night a seeded harness replaced it, driving the real room engine; by 2026-09-19 02:00:46Z it caught three deliberately broken engines, each by name. I haven't re-checked it against all eight invariants since. |
| The ticket was burned before the room and seat were checked. Present a valid ticket to a polled room and it's spent on a connection that is then refused 409. Harmless, since the browser mints another, but the wrong order. | Not re-checked for this update. |
One leaderboard game id per process. gameId() reads the deployment tier, so everything a deployment submits goes to the same board. The leaderboard's own production stack didn't exist either, and Figure 5 is from the DEV stack. | The production stack exists. The game-id item was not re-checked. |
| The two-hour soak hadn't finished. At minute 90 of 120 its live counters read 260 games finished, 404 reconnects, 133 leaves, zero protocol violations, zero unexpected closes, zero 5xx, snapshot p95 1 ms, memory 1.56% below baseline. | It finished at 21:53:43Z that night and passed all 14 of its conditions. The soak write-up has the numbers. |
Predictions vs outcome
These are the expectations the design leaned on, not formal predictions, checked against what the record shows.
| What we expected | What happened | Right? |
|---|---|---|
| Pushing the poll's own JSON over the socket would leave the page's rendering untouched | The call sites changed by one word; the realtime browser test passed 2 of 2 in all five final verification runs | Yes |
| The ordered drain would catch finishes an earlier call had missed | One call swept up two older rows, 741.85 s and 915.10 s after their finishes | Yes |
| The soak would stay healthy to minute 120 | It passed all 14 conditions | Yes |
A finished ws game would reach the leaderboard | Badges did; scores reached no board, because no code in the Rust service posted one | No |
| A room owner that holds its room would be quick anywhere | Its 250 ms database timer made the median answer 3,658 ms at 38 ms from the database | No |
| A warm benchmark describes a player's first request | The first live write took 3,265 ms cold, about twenty times the warm median | No |
Key takeaways
- Treat a 1.5-second poll as a real transport, not a placeholder. It's what production still runs; its price is 750 ms of average staleness and traffic that grows with people rather than events.
- Push the same JSON over the new pipe. That's what let the rendering code stay untouched; the call sites changed by one word.
- Fix a room's transport for its whole life. Both services can then assume a single writer, and every unknown value can fail safely to
poll. - Deliver latest-wins, not every version. Over a
watchchannel a slow client skips snapshots instead of queueing them, which keeps memory bounded. - Write the announcement inside the transaction it announces. Then drain it as an ordered log with a committed offset, and stop at the first bad row so the offset stays honest.
- Benchmark at the distance you'll run at. Our owner task was quick beside its database and slow 38 ms away.
What I'd do next
Prove the relay with a real production ws game, since that's the one link between a socket game and a leaderboard that nothing has exercised yet. Then move a small share of new production rooms to ws, with the new readiness check deciding when to fall back to polling.
After that, the two items this update didn't re-check: move the ticket burn to after the room and seat checks, and give the web client a leaderboard game id per game rather than one per process.
Evidence
All times UTC. Paths are relative to each repository.
| Claim | Source |
|---|---|
Poll interval 1,500 ms; the handover branch on transport: "ws" | hearso-web app/room/[roomId]/room-client.tsx (POLL_MS, openSocket) |
| Answer scoring, 10 s limit, 2 s grace; 900 ms → 191 points | hearso-web lib/game.ts; lib/fixtures/state/question-board-on.json |
| Conditional answer write; the state route as the game loop | hearso-web app/api/rooms/[code]/answer/route.ts, app/api/rooms/[code]/state/route.ts |
rooms.transport stamped at creation; absent, null and unknown read as poll | hearso-web lib/room-transport.ts, lib/room-transport-switch.ts, app/api/rooms/route.ts |
| Ticket format, 30 s life, 5 s skew, frozen key order | hearso-web lib/ws-ticket.ts; cutover/WS-CUTOVER-PLAN_1.md C2 |
Ticket route: cookie only, ws rooms only, live seat only, no-store | hearso-web app/api/rooms/[code]/ws-ticket/route.ts |
| Origin → verify → burn → 409 → 403, all before the upgrade; 4,096-byte frames; 4009 on replacement | hearso-rt crates/transport-ws/src/lib.rs |
SET ticket:<jti> 1 NX PX 40001; AUTH_TICKET_REPLAYED | hearso-rt crates/auth/src/lib.rs |
heartbeatMs: 10000 in session.welcome; last 64 command ids replayed | hearso-rt crates/transport-ws/src/connection.rs; hearso-web lib/room-socket.ts |
| One owner task per room, mailbox capacity 32; snapshots published per viewer | hearso-rt crates/bin-server/src/main.rs; crates/transport-ws/src/lib.rs |
Latest-wins delivery (watch + borrow_and_update); versions must increase | hearso-rt crates/transport-ws/src/wire.rs; hearso-web lib/room-socket.ts |
Answer command: {index, choice}, stale index refused 409, FOR UPDATE + conditional UPDATE | hearso-rt crates/rooms/src/commands.rs, crates/persistence/src/answers.rs |
Snapshot is the state route's payload plus "transport":"ws" | hearso-rt crates/rooms/src/snapshot.rs; hearso-web lib/fixtures/state/*.json; plan D5 |
Command routes answer 409 room_uses_realtime for a ws room | hearso-web app/api/rooms/[code]/{start,answer,advance}/route.ts |
Finish transaction: status CAS, ledger, profiles, plays, one outbox row | hearso-rt crates/persistence/src/finish.rs |
Outbox schema, seq identity, unique (room_code, game_round, finished_at), 7-day prune | hearso-web supabase/migrations/20260918010100_add_websocket_room_transport.sql |
Signed callback: v1=<hex HMAC-SHA256>, 60 s either way, retries at 1 s / 4 s / 10 s | hearso-web lib/internal-callback.ts, app/api/internal/games/finished/route.ts; hearso-rt crates/leaderboard/src/lib.rs |
| Drain in seq order, award then mark, stop at the first failure | hearso-web lib/finish-outbox.ts |
| Drains of 0.12 s and 0.06 s; sweep of two older rows at 741.85 s and 915.10 s | cutover log, web-side entry, 2026-09-18T11:57:48Z |
| Server-only submission; the trust boundary is the deployment boundary | hearso-web ARCHITECTURE.md §3, docs/agent-brief.md §2, lib/leaderboard-client.ts |
Idempotency key trivia-<roomCode>-<playerId> (the code builds it as trivia-${roundId}-${playerId}, and the room flow passes the room code as roundId); default fan-out all-time + daily + weekly; guests scoped to ["daily"] | hearso-web lib/leaderboard-targets.ts, lib/room-flow.ts (roundId: code), lib/profile.ts (GUEST_BOARDS); online-leaderboard src/lib/score-update-command-builder.js |
Lambda nodejs22.x, arm64, 256 MB, 10 s; PAY_PER_REQUEST; TTL on expiresAt | online-leaderboard template.yaml |
Table keys, RankIndex key schema and INCLUDE projection | online-leaderboard template.yaml; docs/adr/0010-table-and-index-design.md |
rankKey shape, SCORE_CEILING 1,000,000,000,000, PAD_WIDTH 13; conditional improvement write | online-leaderboard src/lib/score-key-builder.js, src/constants/score-limits.js, src/lib/score-update-command-builder.js |
leaderboardId = <gameId>#<variant>#<period> | online-leaderboard src/lib/leaderboard-id-parser.js |
| Retention 35 / 400 / 400 days; all-time never expires | online-leaderboard src/constants/leaderboard-retention.js |
Propagation p50 159 / p95 210 / p99 322 ms, 300 samples, DEV us-east-1, 0.00% errors; the per-size table | online-leaderboard benchmarks/20260818T193819.881Z/report.md; online-leaderboard PR #49 |
| First live write 3,265 ms on a cold-start sample | online-leaderboard PR #30 |
| Nothing deployed on 2026-09-18; VPS, TLS and hosted migrations out of scope | cutover/WS-CUTOVER-PLAN_1.md §1 |
No leaderboard submission in the Rust service; submitted false on ws seats | hearso-rt crates/bin-server/src/main.rs (no hearso_leaderboard submission call); cutover log, service-side entry 2026-09-18T11:55:00Z ("Leaderboard target planning/submission remains unimplemented in the Rust crate"), web-side entries 2026-09-18T11:57:48Z and 23:28:25Z |
crates/sim was 43 lines and checked none of the eight invariants | hearso-rt crates/sim/src/lib.rs; plan §8; hearso-ws-game-service PR #1 |
| The harness that replaced it; three broken engines caught by name | cutover log, service-side entries 23:10:00Z and 23:25:00Z (hand-stamped); web-side entry 2026-09-19T02:00:46Z |
| Realtime browser test 2 of 2 in all five final runs | cutover log, web-side entry 2026-09-18T22:15:20Z |
| Soak minute-90 counters; the soak's pass | cutover log, web-side entries 2026-09-18T21:24:25Z and 22:08:52Z |
| The 250 ms database timer, 3,658 ms median on 2026-09-23, 328 ms on 2026-09-27 | hearso-ws-game-service PRs #10 and #11; the benchmark post |
hearso-rt on a VPS; the relay live and not yet proven with a real game; the production leaderboard stack live; new rooms still polled | the overnight handoff for 2026-09-26, entries of 2026-09-26 and 2026-09-27 (06:30Z) and the transport dial at 0 on 2026-09-28 |
| Readiness and a database heartbeat deployed | hearso-ws-game-service PR #22, deployed 2026-09-28 about 19:25Z (overnight handoff) |
Branch 45 commits ahead of develop on 2026-09-18 | git log --oneline origin/develop..HEAD, read 2026-09-18 |