Watch it fail first
Writing tests only after a bug turns out to be the correct policy instead of a discipline problem, and there's a step at the end that almost nobody does.
The same bug turned up twice, about 18 months apart, on the same store. A discount bigger than the order total, a negative amount going out to the payment provider, and an error message that nobody in the building could interpret.
I found the fix from the first time while I was investigating the second time. Same file, my name on it, early last year. The fix was still there and still correct, doing exactly what I’d written it to do. A different code path had grown up next to it and made the same mistake a few inches over, and nothing anywhere in that project knew that a negative total was a thing that must not happen.
The commit message was even pretty good. There was just no test 🙃
Closing the barn door
I’d guess 9 out of 10 of the tests I’ve written that ever caught anything were written in response to a bug that had already happened. Not in advance, not while designing the thing, always afterward. Nothing in the pyramid tells you that, because the pyramid is about what shape a suite should end up and says nothing about what makes anybody write the next test.
For years I read that as a skill issue on my part. I’m meant to be writing tests up front, properly. I don’t, so the after-the-fact ones were what I settled for instead of what I should have been doing.
I’ve changed my mind. I think it’s the correct policy and my mistake was not noticing it was one.
The space of tests you could write is basically unbounded. The space of tests worth writing depends on where the system is actually fragile. You can’t know that in advance, or at least I can’t. You can know where it has already broken, and bugs cluster: the same modules, the same assumptions, the same three or four pieces of logic that were complicated the day they were written and are complicated now.
So a bug that has already happened is the best evidence you’re ever going to get about where to spend testing effort. It’s specific, it’s real, somebody was affected by it, and it costs nothing to obtain because you already paid for it in full.
Which is the part that turns it into a rule you can hold somebody to, including yourself. A bug isn’t finished when the fix works. It’s finished when there’s a test that fails without the fix.
Have you tried turning it off and on again
The order matters more than it sounds like it should.
Reproduce it, which is the first step of any debugging and the one everybody skips, me very much included.
Write the smallest test that exhibits it, and run it before you touch anything else. It has to be red. If it comes out green then you haven’t reproduced what you think you’ve reproduced, and finding that out now costs you a minute where finding it out in three months costs you a lot more than that.
Then fix it, and watch the test go green.
Then revert the fix and watch it go red again.
That last one is the step nobody does. It’s the only thing that proves the test is attached to the fix instead of to something incidental sitting nearby. A test written after a fix that has never been seen to fail might be asserting nothing at all. It will look identical to a real one in the report. Which is a suite nobody believes arriving from the other direction. It’s worse than a gap, because a gap doesn’t tell you you’re covered.
30 seconds. It converts a test you hope works into one you’ve watched work.
What’s in a name?
The name isn’t test_bug_4471, and it isn’t test_discount_fix either.
// A 100% voucher on an order with a delivery charge produced a negative
// total, which the payment gateway rejects with a message nobody can
// interpret. Reported twice, 18 months apart, by the same client.
test('a discount larger than the subtotal reduces the total to zero, not below', () => {
expect(total({ subtotal: 20, delivery: 5, discount: 40 })).toBe(0);
});Two reasons for writing it that way.
The ticket number will not survive. Every project I’ve worked on has changed issue tracker at least once, the numbers never transfer, and a test named after a ticket eventually becomes a test named after nothing. That’s the same argument I made about leaning on a ticket link in a commit message years ago and it keeps being true, which is either vindicating or depressing depending on the day.
And the name is what somebody reads at 5 pm when the build goes red. test_bug_4471 failed tells them nothing, so now they’re opening files. “A discount larger than the subtotal reduces the total to zero, not below” tells them which promise they just broke and lets them decide whether they meant to break it.
The comment is doing work too. The test file is the one artifact guaranteed to be opened by whoever next touches that code, which makes it a better home for that context than a wiki page or a ticket, and honestly a better one than a commit message, since you have to already suspect something to go looking there.
Ghosts of bugs past
The side effect I didn’t see coming, after a few years of doing this on one long-running project.
The suite is now a list of everything that has ever gone wrong with that system, in roughly the words of the people it went wrong for. It isn’t balanced, there are whole areas of the code it doesn’t touch, and the coverage percentage is nothing to write home about.
It’s an honest document though, in a way a coverage-driven suite never is. Every test in it is there because something happened to somebody. Reading it is the fastest way I know for a new person to find out where the gotchas are. It’s a much better answer than “read the code and be careful,” which is what I used to tell people.
And with the usual caveat that I haven’t measured this properly and I’m one person on a handful of projects: bugs that got a regression test don’t come back, and bugs that got a fix without one come back at about the rate you’d sadly expect.
Some things only fail once
Being fair to it, because “only write tests after bugs” would be a silly absolute.
It’s reactive by definition, so the first occurrence always gets through. For a brochure site that’s a trade I’d take. I’ve argued for it before. For anything where failing once is unacceptable it isn’t, and that list is payments, authentication, anything that deletes data, anything holding personal information. Those get tests written in advance, and the useful part is that the list is short enough to fit on one page, so writing it down is an afternoon and not a program of work.
So the honest version is two-tier. A small, deliberately chosen set of things tested before launch because one failure is too many, and then everything else tested reactively, from evidence, as it earns it.
40 minutes, 18 months late
I wrote the test a few weeks ago. It’s 6 lines including the comment and it took about 40 minutes, nearly all of which was the reproduction.
The bug isn’t what stays with me. It was fairly ordinary as bugs go. It’s that 18 months earlier I’d found it, understood it, fixed it properly and written a decent paragraph explaining myself, and then filed every bit of that somewhere that can’t run. A commit message can tell you what happened. It can’t stop the same thing from happening again. I’d spent a long time treating it like it could.