How FlowChassis Was Tested With 240 Worker Kills and Not One Lost Order
FlowChassis makes one main promise: whatever happens to the process, every run picks up where it stopped, and nothing it did is lost or done twice without warning. The only honest way to test that promise is to break the process, over and over, at moments nobody chose, and then check the results against something the engine can’t fake.
So that’s what the tests do. This post goes through them, with the numbers from the recorded results in the code repository. Every figure below comes from those files.
The Crash Demo
The main test is a small online shop. Each order goes through five steps: reserve the stock, charge the card, write a receipt, ship, notify the customer. Reserve, charge and ship have undo steps. The receipt doesn’t, so compensation has to pass over it.
Every step writes its effect to a ledger file before it returns, with one line per effect, flushed to disk. The ledger stands in for the outside world, the payment provider and the courier. It’s the part the engine doesn’t control, which is what makes it a fair witness.
Two kinds of trouble are built into the orders. About one in four has its shipping refused, and must end compensated: charge refunded, stock released. About one in five has a card network that times out on the first two attempts, so those runs must retry.
A supervisor starts a worker process, waits a random 5 to 120 milliseconds, and kills it with SIGKILL, which gives the process no chance to tidy up. Then it starts it again. Each new worker opens the journal, starts any orders not yet started (starting an order is idempotent, so the rest are just found again), and carries on. After the planned number of kills, the supervisor lets a worker finish.
The Results
The recorded runs at two sizes:
| Default demo | Large demo | |
|---|---|---|
| Orders | 200 | 1,000 |
| Orders in flight at once | 4 | 16 |
| Worker processes killed | 40 | 200 |
| Step attempts a kill cut off mid-step | 136 | 1,747 |
| Undo attempts a kill cut off mid-undo | 14 | 173 |
| Completed | 149 | 750 |
| Compensated | 51 | 250 |
| Lost, stuck or failed | 0 | 0 |
| Problems found by the checks | 0 | 0 |
Recorded with FlowChassis 0.1.0 on a 2-CPU Linux cloud machine.
After the last worker exits, seven checks run against the journal and the ledger. Every order started, appears in the journal once, and finished. Each ended the way its own rules said: shipped orders completed, refused ones compensated. Every line in the ledger belongs to an attempt the journal recorded, with the same step and attempt number, and no attempt left more than one. The ledger and the journal agree on what each order did and undid. Attempt numbers only go up and stay within the limit, and no step runs again after failing for good: refused shipping attempts leave a trace in the ledger before they fail, so a second try would show. Steps happened in flow order, and undos newest first, each after its step.
And every repeated effect is accounted for. In the large demo, 46 of 5,296 effects happened twice. The check traces each one to an attempt that a kill cut off after its effect reached the ledger but before its checkpoint reached the journal, so on restart the step ran again. That’s at-least-once execution working as designed, and it’s why every step gets an idempotency key: an outside service given the same key twice does the work once.
Torn Writes
A kill can’t tear a write in half, but a power cut can. To cover that, a test writes a journal, then cuts it at every byte of its last record in turn: one byte in, two bytes, and so on to the end. Every one of those cut journals has to open, drop the torn record, show the run as it was before it, and then finish the run normally. Every one did. The bytes cut off aren’t thrown away: they go into a file beside the journal.
A tear only counts as one when nothing whole follows it. A flipped byte, a bad checksum or an impossible length anywhere earlier in the file makes the engine refuse to open the journal, and leave it untouched, rather than skip a record and everything after it. The flowchassis check command reports where the damage is, and its exit status says which of the two it found.
A Full Disk
What should an engine do when it can’t write its journal? It should stop, because a step it can’t record is a step it can’t recover. A test gives the engine a store that starts failing in the middle of a run. The engine stops before the next step, reports why, and runs nothing else. When the journal is opened again with a working disk, the run resumes from the last good record and completes.
A Second Pair of Eyes
Before the first release, an independent review read the code against its own promises and tried to break it. It found real defects in that draft. The worst: canceling a run at the same moment as compacting the journal could crash the program and leave a journal that wouldn’t open. Others were subtler. A crash in one narrow gap could make a step that had failed for good run again. A process that crashed inside the same step every time could loop forever. And a single flipped bit in an early record could look like a torn write and cut off the rest of the file.
All of them are fixed, and each one has a test that reproduces it. A second review then went over the fixes themselves and found more problems in them, among them a cancel that could slip in just before a run ended and be ignored. Those are fixed and tested too. The crash demo got stricter too. Its old checks couldn’t see two of those defects; the new ones do. To prove it, the step-runs-again defect was planted back into the engine on purpose, and the demo reported it 27 times in one run.
Everything Else
The library and the command have 73 test functions, covering every call and every status: retries and their growing waits, permanent errors, panics in steps, cancel during a step and during a retry wait, compensation resumed after a restart, compaction in the middle of a compensation, many callers starting the same order at once, and group commit, where records written at the same moment share one flush to disk.
Each of the four example programs behind the case studies has its own test too. The video transcoding one kills the program with SIGKILL partway through and checks that the second run carries on at the segment it was on.
All of them also run under Go’s race detector on Linux. They run on Linux, macOS and Windows on every change, and each release’s binaries run the crash demo on their own kind of machine before the release goes out.
Speed
Every checkpoint is on disk before its step moves on, so speed depends on how fast the disk can flush. On the small cloud machine the results were recorded on, a flow of three empty steps ran at about 1,800 steps a second with one worker, 6,900 with eight and 26,000 with 64. The numbers grow with concurrency because runs that write at the same moment share a flush. Real steps call real services and take far longer than an empty one, so for most programs the journal won’t be what they wait on.
The demo is in the command-line tool. Download it and run flowchassis demo to watch it break things and check them on your own machine.