Flaky tests usually aren't
Calling a test flaky is a decision to stop looking, and the re-run button does more damage than the test ever did.
The checkout test failed about 1 run in 15.
Nobody had investigated, because everybody already knew what it was. It’s flaky. You click re-run, the job goes green, you move on. That had been the arrangement for about eight months and there was a shared, almost fond exasperation about it, the way you’d talk about a faucet that drips.
I looked at it eventually, because I’d run out of more urgent things to ignore.
It was a real race. Two requests fired off the same interaction with nothing ordering them, and when the slower one came back last the page rendered stale state. The test caught it 1 run in 15 because that’s roughly how often the network happened to misbehave in the right direction. About 40 minutes to find, nearly all of which was getting it to happen on purpose.
Which means the site did it too.
About 1 customer in 15, at that step, was being shown the wrong thing. None of them reported it and I wouldn’t have either. It looks like the site glitched, you press back, and it’s fine.
So for eight months the only thing in that building filing a bug report about it was the test, and every one of those reports got closed with a click.
The dog that did bark
That’s the part I’d like to plant, so let me be tedious about it.
Calling a test flaky is not a diagnosis, it’s a decision not to investigate. It sounds like a property of the test. So it sounds like something you found out, and it’s actually something you declared so you could stop.
A test that fails intermittently has discovered non-determinism, and non-determinism doesn’t come from nowhere. Something in that system produces different answers on different runs. That something is in production too, where nobody has a re-run button.
Martin Fowler wrote the definitive version of this back in 2011, and I’d read it years before any of this happened, which is the part I find least flattering. Knowing an argument and using it turn out to be pretty different skills.
The usual suspects
Five things, in my experience, and only one of them is the machine’s fault.
There’s a sleep in it. Somebody wrote a 500 ms wait, it worked on their laptop, and in it went. That’s a bet that the operation finishes in half a second, and on a build agent running three other jobs it doesn’t, and the fix everybody reaches for is a longer sleep, which is the same bet at longer odds. Wait for the condition instead of the clock.
// waits for the clock, and hopes
await page.waitForTimeout(500);
expect(await page.textContent('.cart-total')).toBe('$42.00');
// waits for the thing you actually care about
await expect(page.locator('.cart-total')).toHaveText('$42.00');Tests depending on each other. They pass in the order you wrote them and fall apart the moment the runner shuffles them or runs them in parallel, because one is leaving state behind that the next one quietly relies on. Turning on random ordering surfaces every one of these in an afternoon, and oof, it is a thoroughly unpleasant afternoon. Worth knowing that this one is usually telling you something about the application as well, since the shared state the tests are tripping over tends to be shared state in the code.
A genuine race in the app, as above. Two requests with nothing ordering them, a click handler that runs before initialization has finished, a subscription that arrives after the component is gone. Canceling requests that are already in flight turns out to be this exact bug wearing a different hat.
Time. Tests that fail at midnight, or on the first of the month, or on the first Sunday in November when the clocks fall back and an hour happens twice. Every one of those is a real bug in how the application handles dates, sitting there waiting for a customer in a timezone nobody thought about. This category is my favorite, because the test is being so specific about what’s wrong and gets ignored anyway.
Animation. The test clicks where the button is going to be. Turning transitions off in the test environment is a completely legitimate fix instead of a cheat, and the hook is already there if you’ve honored prefers-reduced-motion, which you should have anyway.
Press to make green
The failure isn’t the flaky test. Plenty of suites have one. The failure is what grows around it.
Once “re-run the job” is normal, the suite has stopped being a signal and become a formality. And it doesn’t stay contained to the one test, because the habit generalizes. Red, re-run, green, merge. Within a few months nobody is reading a failure, they’re clicking past it. Which is where an editor with every warning light lit ends up as well, by the opposite route, since a signal you always dismiss stops meaning anything about as fast as one that never turns off.
At which point a genuine regression gets exactly the same treatment. It fails, gets re-run, fails again, and now somebody finally looks, having burned half an hour and possibly merged something else in the meantime. The entire value of the suite was that red means stop, and it’s been spent on a dripping faucet.
I’d put it strongly. A suite nobody believes is worse than no suite at all, because it costs the same to run, it costs more to maintain, and it hands you confidence you haven’t earned. At least a team with no tests knows it has no tests. I’d argue for writing very few tests and choosing them carefully. This is the other half of that, since a small suite only works if every red in it means something.
Quarantine, not amnesty
The policy that’s worked for me. It’s a compromise instead of a principle, because sometimes you genuinely do have to get a release out today.
Move the offender out of the blocking suite into a quarantined list. It still runs, it still reports, it just doesn’t block the merge. It gets a ticket and somebody’s name against it.
Two things keep that from being a euphemism for ignoring it: (1) the list is visible, so the number of quarantined tests is a figure everybody can see and a growing one is a conversation, and (2) each entry carries a date, so nothing sits there for eight months without somebody being asked about it.
The other half is keeping evidence. Most intermittent failures are undiagnosable after the fact because nobody kept anything. A screenshot at the moment of failure, a copy of the DOM, the network log, the console output, all saved off the CI run as artifacts.
- name: Keep the evidence
if: failure()
uses: actions/upload-artifact@v3
with:
name: playwright-report
path: |
test-results/
playwright-report/
retention-days: 30It took me longer to remember the syntax than to write it. It’s the difference between “it does it sometimes” and an actual investigation. Without it you’re asking somebody to debug a thing that has already finished happening on a machine that no longer exists.
In fairness to the build agent
Being fair, since I’ve been fairly absolutist so far.
Some intermittent failures really are the environment. A shared build agent under load from everything else in the queue. A container with less memory than anyone specified. A test that hits a real external service, which is measuring the internet instead of measuring your code. A browser that occasionally takes forever to start.
Those are real, they’re worth fixing at the infrastructure level instead of in the test, and they’re a minority. My rough sense, and I’m one person on a handful of projects, is that when somebody actually sits down with a flaky test for two hours there’s a real bug at the bottom of it more often than not. The environmental ones also tend to announce themselves, because they take the whole suite down with them instead of picking on the same file every time.
I click it too
8 months, 1 run in 15, about 40 minutes to find once anybody looked.
I’d like to tell you I’m the person who always looks, and I’m not. I’ve clicked re-run more times than I could count, usually around 5:50 pm, usually because the alternative was staying 🫠 So the realistic ask isn’t never do that, and a policy that depends on everybody being rigorous at the end of a long day isn’t a policy, it’s a wish.
It’s that the click leaves a mark somewhere. A counter on the job, a line in a log, anything that accumulates. Because the reason nobody connected those two facts for eight months wasn’t that they’re hard to connect, it’s that clicking a button leaves no trace, so there was never a moment where the pattern sat in front of anybody. The story the team told itself about that test got to stay true for exactly as long as nothing was writing it down.