The Invariants I Didn't Hold
An honest accounting of a write-ahead log: what it guarantees, where the documentation outran the code, and the pattern in every gap.
#reliability
11 posts
Testing a Thing Whose Whole Job Is Surviving a Crash
You can't unit test a power cut. What I built instead, and the gap I left in the middle of it.
Three Ways to Recover, and What a Torn Tail Costs
Recovery isn't about salvaging as much as possible. It's about knowing exactly where you stopped believing the file.
Segments, and the fsync That Hit the Wrong File
Splitting the log into files makes almost everything easier. The seam between writing and rotating is where a durability bug was hiding.
Backpressure, and a Cancel That Doesn't Cancel
A bounded queue forces you to decide who waits. Cancellation forces you to decide what an error means.
The Reliability Envelope I Didn't Finish
An honest accounting of what this job queue guarantees, what it doesn't, and the config settings that do nothing at all.
What 'Processing' Means When the Worker Is Gone
A status column records what a job was doing. It cannot record whether anyone is still doing it.
A Timeout Doesn't Stop the Work
Deadlines protect the worker, not the job. Once blocking code is involved, the difference stops being academic.
Retries, Backoff, and Why Jitter Is Not Optional
Retrying is easy. Deciding when to retry, and making sure everyone doesn't decide the same thing, is the actual work.
The Server Forgets Everything and That's Fine
Rooms live in memory and get thrown away. The client is the persistence layer, which changes what reconnecting means.
What Happens When One Client Can't Keep Up
Every broadcast loop contains an assumption that send() works. Backpressure is what you build when it doesn't.