The Invariants I Didn't Hold
An honest accounting of a write-ahead log: what it guarantees, where the documentation outran the code, and the pattern in every gap.
#backend
23 posts
Testing a Thing Whose Whole Job Is Surviving a Crash
You can't unit test a power cut. What I built instead, and the gap I left in the middle of it.
Checkpoints, and What the Benchmark Wasn't Measuring
A 2.55x speedup that was real, produced by a mechanism that doesn't exist, hiding a bug that resurrects deleted keys.
Three Ways to Recover, and What a Torn Tail Costs
Recovery isn't about salvaging as much as possible. It's about knowing exactly where you stopped believing the file.
Segments, and the fsync That Hit the Wrong File
Splitting the log into files makes almost everything easier. The seam between writing and rotating is where a durability bug was hiding.
Backpressure, and a Cancel That Doesn't Cancel
A bounded queue forces you to decide who waits. Cancellation forces you to decide what an error means.
Group Commit, or Refusing to Pay for fsync Alone
Getting from 255 writes a second to nearly ten thousand, without weakening a single durability guarantee.
The Sequence Number That Was Never Written Down
The LSN is the identity of every record in the log. It isn't stored on disk, and three problems follow from that.
Twenty-Two Bytes in Front of Every Record
A length prefix tells you where a record ends. A checksum tells you whether to believe the length prefix.
What a Write-Ahead Log Actually Promises
The durability primitive underneath every database, and the one number that shapes every decision in it.
The Reliability Envelope I Didn't Finish
An honest accounting of what this job queue guarantees, what it doesn't, and the config settings that do nothing at all.
Tracing a Job That Outlives Its Request
Parent and child is a claim about containment. Queued work isn't contained by anything, so it needs a different relationship.
The Load Test That Was Supposed to Fail
173,000 requests with zero errors told me nothing useful. The test that broke its own thresholds told me everything.
What 'Processing' Means When the Worker Is Gone
A status column records what a job was doing. It cannot record whether anyone is still doing it.
A Timeout Doesn't Stop the Work
Deadlines protect the worker, not the job. Once blocking code is involved, the difference stops being academic.
Idempotency Is a Race Until the Database Says Otherwise
Check-then-insert looks correct and isn't. The only reliable fix is to stop checking and let a constraint decide.
Retries, Backoff, and Why Jitter Is Not Optional
Retrying is easy. Deciding when to retry, and making sure everyone doesn't decide the same thing, is the actual work.
Three Priority Tiers in One Number
A single Redis sorted set gives strict priority and FIFO inside each tier. Then I checked the arithmetic.
Two Stores, One Job, and the Gap Between Them
Splitting durable truth from fast dispatch buys a lot. It also creates a dual write I never made atomic.
Why I Didn't Just Use Celery
A job queue is a promise that work you accepted will eventually happen. Making that promise honestly is the hard part.
The Server Forgets Everything and That's Fine
Rooms live in memory and get thrown away. The client is the persistence layer, which changes what reconnecting means.
What Happens When One Client Can't Keep Up
Every broadcast loop contains an assumption that send() works. Backpressure is what you build when it doesn't.
Writing a Binary WebSocket Protocol by Hand
What actually goes over the wire in a collaborative canvas, and why the ready made library didn't fit.