Streaming live prices to 50K users on a $200/month budget

Part 2 of a 4-part series on engineering FTL, a live football trading platform. Published with the client’s permission. The production codebase is proprietary. A public skeleton of this component (ws-fanout) is in preparation.

Picture a stadium of phones, all waiting for the same price to twitch. A goal goes in. Fifty thousand screens should light up together. That is FTL’s ticker: a virtual match-day feed. In-game coins, a $200 Azure credit, prices that are allowed to drop.

This post is how we sent those prices without melting the bill. At the end I wrote two short “what I’d do next time” notes: one for this game, and one if a print were real money.

One morning during tournament prep, staging was burning ₹1,234 a day. Roughly eight times the budget. Every dashboard looked healthy. The culprit was quiet: our Container App was in multiple-revisions mode, so every nightly scale-down/up cycle (az containerapp update) created a new revision without turning the old one off. Nine revisions, each keeping its minimum replicas warm. Ten always-on vCPUs, for nobody. We caught it in a cost audit. The fix was one line of az-cli plus a revision-mode switch.

I tell that story first because it frames everything below. At this budget, every byte and every millisecond is a line item. The question this post answers: how do you design a WebSocket fan-out layer for 50K concurrent users when your total infrastructure budget is a $200/month Azure credit?

Lossy is a choice. A price tick is 100ms old in 100ms. We designed for that.

Start with the wire budget

The biggest cost in real-time systems isn’t the computer thinking. It’s the computer talking. Bandwidth. So we started from bytes, not code. The price update payload:

type PriceUpdate struct {
    InstrumentIdx uint16  `msgpack:"i"` // compact instrument index
    Price         float64 `msgpack:"p"` // current price
    Source        uint8   `msgpack:"s"` // live / trade / postmatch / replay
}

Three fields, msgpack-encoded to ~21 bytes. Two tricks make it that small:

1. The uint16 instrument index. Sending the instrument UUID costs 36 bytes. That’s writing the player’s full name on every scoreboard update. Instead, every instrument gets a uint16 index at registration; a Redis hash (instrument_idx) maps UUID → idx, and clients resolve it once at connect. A jersey number. 2 bytes instead of 36, on every message, to every client, forever.

2. Single-byte tags. msgpack field names repeat on the wire. One-character names (i, p, s) cost 1 byte each instead of 5–10. A uint8 source code instead of a string enum.

Then we budget the true wire cost, which includes the stuff nobody puts in the slide deck:

msgpack payload          ~21 B
WebSocket frame + TLS + TCP/IP + alignment   ~70 B
─────────────────────────────────────────────
total per broadcast      ~95 B

At 50K users × 10 subscriptions each, that 95-byte number is the input to every bandwidth and egress decision we make. Which is why this exists:

// hub_test.go
func TestPriceUpdateWireSize(t *testing.T) {
    data, _ := msgpack.Marshal(PriceUpdate{InstrumentIdx: 1, Price: 12.34})
    if len(data) > 25 {
        t.Fatalf("wire budget blown: %d bytes", len(data))
    }
}

A regression test on wire size. Anyone can add a field to a struct; the test makes adding a byte to every message sent to every client a failing build. This is the cheapest performance tool I know: encode your budget as a test, because humans don’t re-read cost spreadsheets, but CI never forgets.

The broadcast path: from one goroutine to a worker pool

Our first broadcast loop was the obvious one. One goroutine per replica, iterate subscribers, write:

for _, conn := range conns {
    writeWithDeadline(conn, websocket.BinaryMessage, data)
}

The writeWithDeadline wrapper (100ms write deadline, evict on failure) stops one dead TCP client from stalling the loop forever. But at scale the math stopped working. Single-goroutine write throughput is roughly 10,000 writes/sec at 0.1ms per write. Fine for 2 simultaneous matches at 5K users. At 4 matches broadcasting every second, the drain time per broadcast call exceeded the inter-call window. Broadcasts started stacking.

Think of one usher trying to tap 5,000 shoulders. We split the crowd.

The fix was a W=16 worker pool with stable per-connection routing: each connection is assigned one worker at AddConn time (connWorker map[*Conn]int) and never reassigned. All writes to a connection come from the same goroutine, which means no write mutexes at all.

broadcast call → channel per worker → worker drains its connections
W=16, buffer=64 per channel
drain per worker at 5K users, 4 matches: ~31ms < 48ms window ✓

Two bugs we caught in review before this shipped. Both would have been ugly in a packed stadium.

Bug 1: the concurrent-write race. The first implementation assigned workers as i % W on the subscriber index. If a connection’s index changed between iterations (it can; subscriptions change), two workers could hold the same connection, and two goroutines could write to one TCP socket concurrently. Gorilla panics on that. The fix: routing keyed on the connection itself, assigned once, never recomputed. i % W distributes load but assumes indices are stable. They never are.

Bug 2: the ping mass-eviction. The heartbeat path (30s ping / 45s timeout) initially shared the worker’s exit condition: one heartbeat failure and every worker terminated, evicting all connections at once. A one-second Redis hiccup would have looked like a full outage. The fix split the (enqueued, registered) lifecycle so a ping failure evicts one connection, not the fleet.

Backpressure policy: when a worker’s channel is full, we drop the message (logged, counted in ws_broadcast_drops_total) rather than block. A price tick is 100ms old in 100ms; a stalled broadcast queue is worth less than a dropped tick. Latency-sensitive broadcast is one of the rare cases where lossy is correct.

Adapt the cadence, not just the code

The other 90% savings came from noticing that not every second is equally valuable. Football has a phase structure, so the broadcast cadence follows it:

• Pre-match, idle periods: 10s between ticks
• Ball in play: 2s (Sportmonks feed cadence)
• 30 seconds after a goal, the peak-trading moment: 1s

Every 100ms of cadence has a dollar figure attached, computed from the 95-byte wire budget. Combined with delta-gated publishes (don’t send if price didn’t change) and 250ms coalescing windows, the cost of real-time stops being linear in users.

One infra trap worth the price of admission: Azure Container Apps WebSocket connections are sticky, so scaling events force reconnects. Our scale-up script warms WS replicas 15 minutes before kickoff, scheduled against the fixture calendar, not reactive to load. Reactive autoscaling on a live sports feed is reactive to something that already happened.

Testing a fan-out layer you can’t load locally

The test tooling taught us its own lesson: k6 can’t decode msgpack, so a k6 connection soak validates establishment (we ran 5K-connection and spectator scenarios) but not payload correctness. The real broadcast soak is a Node.js script that connects, decodes every frame, and asserts ≥99% valid msgpack decode with zero errors over 120s. “The socket is open” and “the client is receiving sane data” are different failures, and only one of them shows up on your users’ screens.

The full stack, at budget

For the cost-curious, the whole platform (2 API replicas, 3 WS replicas, a flusher worker, Postgres Flexible Server, Azure Redis, Container Apps environment) runs on roughly $1,200/month at full design capacity, and dramatically less with the nightly scale-to-zero scripts stretching the $200 free credit. For contrast, the serverless first draft (Azure Functions + Cosmos DB + Service Bus + Web PubSub) priced at $2,500/month before the Web PubSub bill alone ($1,200/mo for 50K connections) made it unviable. The rewrite wasn’t about languages. It was about who charges you for fan-out.

What I would do differently for this use case

Same stadium. Same 50K phones. Same $200 credit. Same permission to drop a tick. Re-reading this for a public ws-fanout skeleton, I’d keep lossy. I’d tighten four things.

One envelope, many prices. After a goal, a lot of footballers move at once. We still send one PriceUpdate per instrument, and pay the ~70-byte envelope every time. The 95-byte slide-deck number times 10 subscriptions is the expensive version of that. I’d pack every dirty instrument for that connection into one binary frame inside the 250ms window we already have, and count egress as 50K × (21×subs + ~70) per interval. If a scaled uint32 price fits in the 25-byte test, that’s a free field. Keep lossy. The wire-size test stays. Anyone can add a field. CI makes adding a byte to every phone a failing build.

The 50K math is still open. The number we published is ~31ms drain, under a 48ms window, at 5K users, 4 matches, W=16. That’s every connection on a replica, not “10 subscriptions.” Three replicas and 50K phones is a story this post does not close. If 0.1ms per write still holds, that’s about 16.7K connections per replica and about 104ms to touch all of them. Still outside 48ms. If people really subscribe to ten footballers, a per-worker index of instrument → connections is how a tick to 10% of users costs 10% of writes. I’d pin W or replica count to a test that fails when drain misses the window. Inter-replica delivery is part 1’s PUBLISH. This post never named a subscriber group. Subscribe and heartbeat should not share a socket’s lifetime, so a Redis blip cannot look like the stadium went dark.

Drop on full. Don’t drop in silence. Channel full means drop, log, bump ws_broadcast_drops_total. A price tick is 100ms old in 100ms. That policy is right. The payload has no sequence number. Idle cadence is 10s, so a silent drop is a 10s lie on the glass. I’d spend leftover room in the 25-byte ceiling on a per-instrument uint32 seq. On connect, and on a gap: an HTTP snapshot of the ten subscribed prices. They already hit Redis for instrument_idx. Alert on drop rate in the 1s window, not just the counter. Soak until decode errors are zero and drops have a bound.

Scale without kicking everyone out. Azure Container Apps stick a WebSocket to a replica, so a scale event is a reconnect storm. The 15-minute fixture warmup only covers a kickoff we already knew about. Multiple-revisions was a cost bug in staging. Mid-match it is an outage. I’d drain connections with a jittered “reconnect” frame, and never az containerapp update in multiple-revisions during a live fixture. Same calendar as warmup. One retry on a 100ms write, then evict: stadium Wi-Fi should not dump the session, and should not look like a disconnect that fires the margin replay. Cap accept rate in a storm. Redis subscribe must restart without closing the replica’s sockets.

What stays: the ₹1,234/day catch, the 25-byte CI bomb, routing on the connection instead of i % W, splitting the ping lifecycle, owning fan-out instead of paying Web PubSub, and warming replicas 15 minutes before kickoff. The pattern is the same as the rest of this series: invisible at a 5K “did the socket open” soak, page one of the postmortem at 50K on goal cadence. I’d freeze instrument_idx as append-only for a tournament, put a mapping version on hello, and turn the two review-caught races into tests so review is not the only net.

What I would do differently for a fintech

Same plant, except a print or a fill is a legal event. I’d keep a fast, lossy ticker for public prices. I’d change five things.

1. Two buses

This plant is a stadium ticker: one PriceUpdate, one worker pool, drop-on-full for everything on that path. For FTL, lossy is correct. A price tick is 100ms old in 100ms. Source: trade is a uint8 origin on a price struct (live / trade / postmatch / replay), not a fill, not an ack, not drop-copy. Fills return on the HTTP path in part 1. I would not put legal events on this socket.

For cash, public market data may be lossy and conflated. Private order events are durable, sequenced, retried.

What I would do: two buses. Isolate worker pools so a quote write cannot evict an order-event session. One pool carries every PriceUpdate today; that is the mixed-criticality line I would refuse for a venue.

2. Sequence, snapshot, gap fill

For FTL, a dropped tick plus four bytes of per-instrument seq and an HTTP snapshot of ten prices is enough to stop a 10s lie on the glass. That is a game recovery. For cash, delivery is a recoverable stream: every increment has a sequence; a gap is a protocol event, not a support ticket.

What I would do: versioned payload with seq and exchange time. Logon snapshot, then increments. On gap, re-snapshot or replay from the log. Delivery SLO is zero silent gaps, not ≥99% decode. Conflation only where the contract says “latest is enough.”

3. Fair-access clock

For FTL, 10s / 2s / 1s cadence, 250ms coalescing, and delta-gating are dollar figures on the 95-byte wire. W=16 drain order is how we kept writes mutex-free. Sticky ACA is how the $200 credit survives a scale event.

For cash, timestamp at the matcher. Same bytes, same order, every replica. Sticky replicas plus drain order is an accidental fairness lottery. I would not ship that as a venue clock. If the product needs delay, it is uniform.

What I would do: stamp at the matcher; fan out a sequenced stream so every replica releases the same bytes in the same order.

4. Replayable log; WS replicas as stateless projectors

For FTL, three sticky ACA replicas and W=16, with inter-replica delivery implied by part 1’s PUBLISH. Scale is a reconnect. Heartbeats already taught us a shared exit is a plant-wide off switch. Isolate them from the write pool so a missed ping closes one session, never the plant.

For cash, a sequencer/WAL sits in front of the sockets. WS replicas are stateless projectors: they subscribe by offset, they restart and catch up. Redis, if it remains, is a last-value cache for snapshots, not the only copy of the stream.

What I would do: scale adds projectors; it does not require 16.7K reconnects to keep seq N+1.

5. Sessions, entitlements, resume

For FTL, connect resolves instrument_idx; ten subscriptions are a bandwidth assumption; warmup is 15 minutes before kickoff. Scale-in is a reconnect storm we scheduled around.

For cash, authenticate; entitle subscribe; user-scoped reliable channels versus public MD; the client presents last seq.

What I would do: treat scale-in as drain + resume, then add or remove projectors. Test the resume path, not only that the socket came up.

The two designs, side by side

Decision	This game	If it were real money
What the socket carries	Lossy price ticks (live/trade/postmatch/replay on one struct)	Public MD and private order events on separate buses
Delivery	Drop-on-full, no seq	Sequenced stream; snapshot + gap fill; fills never silently dropped
Cadence / fairness	10s / 2s / 1s; 250ms coalesce; worker drain order unmeasured	Matcher clock; uniform release; bounded inter-replica skew
Fan-out	3 sticky ACA replicas, W=16, implied part 1 PUBLISH	Stateless projectors on a replayable sequenced log
Connect / scale	instrument_idx at connect; 15 min warmup; scale = reconnect	Authn + entitlements; resume from seq; drain then add/remove projectors
Payload / test bar	~21 B msgpack, uint16, float64, 25 B CI bomb; k6 5K; ≥99% / 120s	Versioned binary, venue id, integer/decimal price, seq, time; 100% decode at design density

This is a high-performance stadium ticker for a virtual market where 50K phones and a $200 credit are the product. The same shape, with two buses, a sequenced recoverable stream, a matcher clock, stateless projectors on a replayable log, and entitled resume, is what I would take into a cash venue.

I’m open to backend/Go roles in fintech. If your product has a real-time firehose and a budget, I’d love to talk. LinkedIn

Resources

• Series: Part 1: atomic trades · Part 3: write-behind · Part 4: margin engine
• Studio: xpar.in
• Code: a public ws-fanout repo is in preparation. The production FTL source is proprietary, so it is not linked here.

Next: the write-behind flusher. How Redis trades stay fast while Postgres stays truthful, and the 4,364 ticks we dropped learning it.