Redis for speed, Postgres for truth: building a crash-safe write-behind layer

Part 3 of a 4-part series on engineering FTL, a live football trading platform. Published with the client’s permission. The production codebase is proprietary. A public skeleton of this component (write-behind) is in preparation.

Think of Redis as the whiteboard the whole room can see, and Postgres as the bound ledger in the back office. Trades happen on the whiteboard. Every 100ms a clerk photocopies the “changed” stickies and copies them into the book. This layer is for FTL: a virtual match-day market. In-game coins, footballer shares, Redis on the hot path, Postgres out of the trade.

The story below is how we stopped losing stickies. At the end: what I’d change for this game, and what I’d change if the coins were real.

During a week of investigating “staging feels slow,” we found a quietly horrifying line in an audit: 4,364 price ticks had been silently dropped by our persistence layer. No error, no alert. The chart had gaps; the ledger had holes; the dashboards were green.

That number is why this post exists. The write-behind pattern (fast writes to a hot store, async persistence to the source of truth) is one of the oldest patterns in systems engineering, and the gap between “I have a background job that syncs to Postgres” and “I have a crash-safe write-behind layer” is exactly where those 4,364 ticks fell.

The architecture

Every trade runs as an atomic Redis Lua script (part 1 of this series) and touches three kinds of state: instruments (price, net shares), wallets (balance), and positions (shares, avg price). After mutating any of them, the script does one more thing:

redis.call('SADD', 'dirty:instruments', instrument_id)
redis.call('SADD', 'dirty:wallets', user_id)
redis.call('SADD', 'dirty:positions', user_id .. ':' .. instrument_id)

That’s the entire durability contract of the hot path: mark it dirty, return, charge on. Postgres never appears in the trade path. A background flusher worker ticks every 100ms and drains the dirty sets into Postgres, which is the source of truth.

Why not write to Postgres synchronously and cache reads in Redis? Because then Postgres is in your trade path under contention, and you’re back to the locking problem we built the Lua engine to avoid. Why not make Redis the source of truth? Because Redis persistence (AOF/RDB) is close-but-not-transactional, and a money ledger deserves a database whose default answer to “did it commit?” is unambiguous. The two-tier design gives each store the job it’s good at.

The hard part, the part that dropped our ticks, is the seam between the two.

Take 1: drain the set, lose the writes

The naive flusher reads SMEMBERS dirty:instruments, writes each member to Postgres, then deletes the set. Three separate failure modes, each one a silent data-loss story:

1. Writes land in the set while you’re draining. Read members → drain → DEL the set → you just deleted IDs that arrived mid-drain.
2. Crash between read and drain. The in-flight members exist nowhere.
3. Partial Postgres failure. If row 3 of 50 fails and you’ve already deleted the set, rows 3–50 are gone.

The fix: RENAME is the whole trick

Our flusher never drains the live set. Instead, each tick does:

RENAME dirty:instruments  dirty:instruments:flushing:<timestamp>

RENAME is atomic in Redis. In one instant, the live set becomes the snapshot, and this is the part to internalize: any writes that land after the rename go into a brand-new live set that this cycle will never touch. Writers were never paused, never blocked, never told to wait. There is no window in which a trade’s dirty-mark can be lost to the flush cycle, because the rename is a single atomic operation and the writer’s SADD is a single atomic operation, serialized by Redis’s single-threaded command execution.

Then the flusher drains the snapshot at its leisure:

snapKey := fmt.Sprintf("%s:flushing:%d", liveKey, time.Now().UnixNano())
f.rdb.Rename(ctx, liveKey, snapKey)
members := f.rdb.SMembers(ctx, snapKey)
// ... drain members into Postgres ...
defer f.rdb.Del(ctx, snapKey)

(An empty live set makes RENAME fail with “no such key”. That’s the normal no-work case, not an error.)

Every failure mode, explicitly owned

A crash-safe layer isn’t one that fails less; it’s one that has decided in advance what each failure means:

Failure	Behavior	Data loss?
Writer crashes after trade, before flush	Trade lives only in Redis; flusher drains it next tick	None (Redis up)
Flusher crashes before rename	Live set untouched, next tick picks it up	None
Flusher crashes between rename and drain	Snapshot key orphaned in Redis	None (see recovery below)
Postgres row write fails	ID re-SADDed into the live set; retried next tick	None
Postgres commit fails at tx end	All members re-queued, not just marked-failed ones	None
Malformed row (e.g. unparseable balance)	Dropped, loudly logged	Accepted (retrying a malformed row forever is worse)

That third row was our 4,364-tick lesson in a previous incarnation. The recovery mechanism is the most elegant four lines in the codebase:

// RecoverOrphanSnapshots runs at startup, before the first flush tick.
keys, _ := f.rdb.Scan(ctx, cursor, live+":flushing:*", 100)
for _, k := range keys {
    f.rdb.SUnionStore(ctx, live, live, k) // merge orphan back into live set
    f.rdb.Del(ctx, k)
}

If the flusher died mid-drain, its snapshot keys are still sitting in Redis. At startup we scan for any *:flushing:* keys and union them back into the live set before flushing resumes. Every orphaned ID gets drained by the next tick. The crash cost a restart, not data.

One subtlety worth stealing: the commit-failure requeue re-queues all members, not just the ones whose individual Exec failed. If tx.Commit fails, every row in the transaction is uncommitted. Re-queueing only the per-row failures would silently drop the rest. This bug is extremely easy to write and produces exactly the kind of gap that surfaces six weeks later as “why does this one user’s balance disagree with the audit log?”

Why sets, not streams or queues?

The obvious objection: “isn’t this a job for Kafka / Redis Streams / a queue?” For this workload, sets are the right primitive, deliberately:

• Sets deduplicate. A user trading 40 times in one tick window adds one entry. The flusher reads the current state from Redis hashes anyway. The dirty set isn’t a queue of events, it’s a queue of keys to re-read. Coalescing by key is the whole win: the flush cost is bounded by state size, not event volume.
• Event-ordering doesn’t matter. Each flush writes the latest value of a key (UPDATE wallets SET balance = $1). There’s no sequence to preserve; there’s only “what is the balance now.”
• At-most-once delivery of events is fine, because delivery is of keys, and the value lives in Redis until drained. The danger window is only “Redis dies before flush”, for which we keep an append-only trade journal in Postgres and can replay recovery from ledger snapshots in 30–60s.

Streams would give us history and consumption groups we don’t need, at the cost of trimming, consumer groups, and backpressure machinery, for a problem (coalescing) that sets solve for free.

Bonus: piggyback the decay

The flusher’s tick loop also drives AMM impact decay. Every N ticks, each instrument’s permanent_impact field shrinks by a fixed rate so an old whale trade’s price footprint fades over hours instead of warping the chart forever. The decayed value is re-SADDed into dirty:instruments, so persistence rides the same cycle. One worker, two jobs, both periodic. Cheap and predictable.

One detail that sounds paranoid until you meet it: the Go decay and the Lua accumulation round permanent_impact with the same formula (floor(x*10000+0.5)/10000), or repeated decay cycles drift by one rounding ULP and the two writers fight forever. Two components writing the same field must agree on rounding exactly, not approximately.

The mental model

If you remember one thing: the dirty set is not data, it’s a pointer. The data always lives in Redis hashes until the moment it’s successfully copied into Postgres. Every design decision above (atomic rename, orphan recovery, re-queue-on-failure) falls out of that one commitment. The moment you start putting data in the dirty structure, you’ve built a queue with a durability problem.

What I would do differently for this use case

Same game, same whiteboard, same 100ms clerk. Re-reading this for a public write-behind skeleton, I’d keep RENAME. I’d change four things around it.

Sticky notes are perfect for “this wallet changed.” They are a terrible film. Forty trades in a second still leave one dirty wallet. The flusher reads the latest balance. That’s the win, and that’s why the sets section above stays. It cannot remember every price along the way. The margin engine walks that film. So I’d keep the sticky notes for wallets and positions, and add a tape for prices: RPUSH or XADD inside the same Lua script that prints the tick. This flusher batch-inserts price_ticks, then trims. Two channels, one worker.

Those 4,364 ticks died on the old drain-then-DEL seam, the photocopy that got thrown away. RENAME closed that hole. Coalescing is a different story: it would still collapse a burst of prices into one “now.”

The published recovery is one Scan with the error discarded. Four lines: COUNT 100, keys, _ :=. Redis SCAN is a cursor. One call is not the whole drawer. I’d loop until the cursor is 0, fail the boot if Scan / SUnionStore / Del errors, and recover all three prefixes (instruments, wallets, positions). The clerk does not start until the photocopies from the crash are back on the board.

Only one clerk at a time. Two processes both renaming, decaying, and deleting the same snapshot is a fair hazard from “ticks every 100ms” plus “drain at leisure.” I’d take a Redis lease (SET NX EX) so only one flusher holds the tick, keep one tick in flight if drain runs long, and Del the snapshot only after every member is committed or re-queued. Not defer Del on panic. Optional: a ver on each hash so a late photocopy cannot overwrite a newer row.

If the whiteboard falls off the wall, the book is behind. Same move as the Lua postscript: pin hashes, dirty sets, and a journal stream with noeviction, and WAIT one replica after the script. Then the part-3 work. The body mentions an append-only trade journal in one sentence. I cannot point to a writer. Second build: XADD in the same Lua as the fill, this flusher drains it. If Redis dies, freeze trading (part 1 already has frozen), hydrate from Postgres, finish SCAN, then unfreeze. A poison row goes to dirty:…:dead with a page, not “Accepted.” Decay belongs in one tiny Lua (HSET + the same rounding + SADD), and I’d skip decay if flush is behind.

Dirty-marks cannot be lost to the flush cycle. Fills can still be lost to Redis. We closed the drain/DEL hole. We did not close Redis death or poison rows.

What I would do differently for a fintech

Same seam, except a “filled” on the phone is a promise about rupees. I’d keep Redis fast. I’d change who is allowed to say yes.

1. The gap is the design

We mark dirty and return. The clerk copies hashes 100ms later. For FTL that’s why a trade feels instant. If Redis dies in the gap, the user saw a fill the book never got. In-game, that’s a ticket.

If this were real money, no 200 until durable media has the row (a ledger insert, or a local fsync). Redis stays the matcher and the read cache.

2. “Balance now” is a snapshot, not a story

Forty fills, one dirty key, UPDATE wallets SET balance = $1. Perfect for “what do they have right now.” A real ledger is a list of events you can point at: this fill, this id, this time. Balances are a sum of those events. Redis can still hold the hot sum. It does not define it.

3. One book

The title says Postgres for truth. During the day the product believes Redis. After Redis death, unflushed fills un-happen. Postgres is the hydrate source. Redis is what the game believes.

I’d pick one commit log. Rebuild the cache from it, freeze until caught up, then serve.

4. Don’t throw away a row because the parser sneezed

The table marks a malformed balance as dropped, loudly logged, accepted. For FTL I’d dead-letter and keep the game up. For real money that key stops, a human looks, nobody “writes it off.”

5. One trade, one posting

Three independent dirty sets can copy a new price without the debit, or the debit without the position. Leaderboard, decay, and the WebSocket plant should listen to the fill. They should not hitch a ride on the durability worker.

The journal is the hot path: wallet, position, and trade capture commit together. Then the flusher is a projector.

The two designs, side by side

Decision	This game	If it were real money
Durability	Redis Lua first; 100ms write-behind	Ledger/WAL commit before 200 OK
What Postgres stores	Coalesced snapshots (UPDATE current hashes)	Append-only events; balances are a projection
Tick / fill history	Second-build tape in Lua for price_ticks; sets for snapshots	Trade-capture and MD journals as sequences
Redis-dies window	Named; WAIT 1 + noeviction + freeze-on-hydrate	Closed: no ack before durable media
Malformed row	Dead-letter + page	Quarantine the key
Source of truth	Redis is live; Postgres is replica + hydrate source	One commit log; Redis is cache/matcher

A match-day spike is the product. The same seam, with a yes that waits for the book, a list of events, one commit log, and one posting per fill, is what I’d build if the coins were real.

I’m open to backend/Go roles in fintech. State reconciliation between a hot store and a durable one is my favorite kind of problem. LinkedIn

Resources

• Series: Part 1: atomic trades · Part 2: WebSocket fan-out · Part 4: margin engine
• Studio: xpar.in
• Code: a public write-behind repo is in preparation. The production FTL source is proprietary, so it is not linked here.

Next, the final part: the margin engine. What happens when a leveraged position should have been liquidated while its owner’s phone was in their pocket.