Most flaky tests aren't
Flaky is a decision, not a diagnosis. The retry button does more damage than the test does.
The checkout test failed roughly one run in 15.
Nobody had investigated because everybody knew what it was: it’s flaky. You re-run the job and it goes green. This had been the arrangement for about eight months and there was a shared, slightly fond, exasperation about it, the way you’d talk about a radiator that needs bleeding.
I looked at it, eventually, because I’d run out of other things to be annoyed by.
It was a real race. Two requests fired on the same interaction, no ordering between them, and when the slower one won the page rendered stale state. The test caught it one time in 15 because that’s roughly how often the network happened to misbehave in the right way.
Which means the site did it too. Roughly one customer in 15, at that step, was seeing the wrong thing, and they weren’t reporting it because it looks like the site glitched and you press back and it’s fine. Nobody in eight months connected the two, because the test result had been given a name that meant “ignore this.”
“Flaky” is a decision
That’s the thing I’d like to plant.
Calling a test flaky is not a diagnosis, it’s a decision not to investigate, wearing the clothes of a diagnosis. It sounds like a property of the test, so it sounds like something you’ve found out, and it’s actually something you’ve declared so you can stop.
"It's flaky" isn't a finding. It's a decision to stop looking, phrased as though it were a finding.
And a test that fails intermittently is a test that has discovered non-determinism. Non-determinism doesn’t come from nowhere. Something in that system produces different outcomes on different runs, and that something exists in production too, where nobody has a re-run button.
The five things it’s usually telling you
There’s a sleep in it. Somebody wrote wait 500ms and it worked on their machine. That’s a bet that the operation completes in under half a second, and on a loaded build agent it doesn’t, and the fix everybody applies is a longer sleep, which is the same bet at longer odds. The real fix is to wait for the condition rather than for the clock: this element visible, this request settled, this state true. Every arbitrary sleep in a suite is an unexamined assumption about timing, and they’re where most intermittent failures live.
Tests depend on each other. They pass in the order you wrote them and fail when run in parallel or shuffled, because one leaves state behind that another relies on. Randomizing the order will surface all of these in an afternoon and it’s a thoroughly unpleasant afternoon. Worth knowing that this one is also telling you something about the application, because the shared state the tests are tripping over is usually shared state in the code.
There’s a genuine race in the app. As above. Two requests with no ordering, a click handler that runs before initialization finishes, a subscription that arrives after the component has gone. I wrote a post in the spring about canceling requests that was, in retrospect, about this exact bug in a different costume.
Time. Tests that fail at midnight, or on the first of the month, or on the last Sunday in October when the clocks go back and an hour happens twice. Every single one of those is a real bug in how the application handles dates, and it’s sitting there waiting for a customer in a different timezone. This category is my favorite because the test is being so specific about what’s wrong and gets ignored anyway.
Animation. The test clicks where the button is going to be. Turning transitions off in the test environment is a completely legitimate fix rather than a cheat, and the hook already exists if you’ve honored reduced motion, which you should have anyway.
The retry button is the actual damage
The failure isn’t the flaky test. Plenty of suites have one. The failure is what gets built around it.
Once “re-run the job” is normal, the suite has stopped being a signal and become a formality. And it doesn’t stay contained to the one test, because the habit generalizes: red build, re-run, green, merge. Within a few months nobody reads a failure, they just click.
At which point a genuine regression gets exactly the same treatment. It fails, gets re-run, fails again, and now somebody looks, having lost 20 minutes and possibly having merged something else in between. And the whole value of the suite, which was that a red build means stop, has been spent.
I’d put it strongly: a test suite nobody believes is worse than no test suite, because it costs the same to run, it costs more to maintain, and it provides confidence that isn’t warranted. At least a team with no tests knows it has no tests.
Quarantine rather than retry
The policy that’s worked for me, and it’s a compromise rather than a principle, because sometimes you genuinely do need to ship this afternoon.
Move the offender out of the blocking suite into a quarantined list. It still runs, it still reports, it doesn’t block the merge. It gets a ticket and a name against it.
Two things make that work rather than being a euphemism for ignoring it. The list is visible, so the number of quarantined tests is a metric everybody can see, and a growing number is a conversation. And each entry has a date, so nothing sits there for eight months without somebody being asked.
The other half is capturing evidence. Most flakiness is undiagnosable after the fact because nobody kept anything. A screenshot at the moment of failure, a copy of the DOM, the network log, the console output, all saved as build artifacts. It costs a few lines of configuration and it’s the difference between “it does it sometimes” and an actual investigation.
The ones that are environmental
Being fair, since I’ve been fairly absolutist.
Some intermittent failures really are the environment. A shared build agent under load from four other jobs. A container with less memory than expected. A test that hits a real external service, which is measuring the internet rather than your code. A browser that occasionally takes three seconds to start.
Those are real and they’re worth fixing at the infrastructure level rather than in the test, and they’re a minority. My rough experience is that when somebody actually sits down with a flaky test for two hours, the majority of the time there’s a real bug at the bottom of it.
Leave a mark
Eight months, one run in 15, and about 40 minutes to find once anybody looked.
I’d like to pretend I’m the person who always looks and I’m not. I have clicked re-run more times than I could count, usually at 5:50, usually because the alternative was staying. The realistic ask isn’t “never do that.” It’s that the re-run leaves a mark somewhere, so that in eight months somebody can see there’s a pattern rather than a radiator.