Debugging is a method

I sat next to a junior in June while they worked on a bug. Two hours, and the process was: change something plausible, refresh, look, change something else, refresh.

Which is what I did for my first four years, and I’d never articulated it as a problem because it does work eventually. It’s just enormously slow, and it doesn’t get faster with experience, because it isn’t a skill, it’s a search.

The thing I realized while watching is that nobody ever taught either of us that there was an alternative. There’s plenty written about debugging tools, and a lot written about incidents after the fact, and almost nothing about the method itself, which is odd for something that’s a genuine majority of the working day.

So here’s mine, such as it is.

Reproduce it first

The first step is to make the bug happen on demand. Reliably, every time, in the smallest number of steps you can manage.

Everybody skips this, because it feels like preamble rather than work, and because you already believe you know roughly where the problem is and want to go and look.

It is not preamble, it’s most of the job. A bug you cannot reproduce is a rumor: you can’t test a hypothesis against it, and worse, you can’t tell whether you’ve fixed it. Half the “fixes” I shipped in my twenties were changes I made to a system that then happened not to exhibit the bug during the 11 seconds I looked at it.

And the act of pinning down the reproduction usually tells you a great deal. “It only happens on the second submission” and “it only happens when the zip code has a space in it” are half the answer, and you get them for free from the process of trying to make it happen twice.

Then make it smaller

Once it reproduces, start deleting.

Remove a component. Remove the stylesheet. Remove half the data. Does it still happen? If yes, that part wasn’t involved. Halve it again.

This is binary search applied to the input rather than to history, which is exactly what git bisect does to a commit log, and it works for the same reason: you don’t have to be clever, you just have to halve the space repeatedly. 20 steps takes you from a million possibilities to one.

The minimal reproduction, when you get there, is usually so small that the bug is simply visible in it. I’d say a third of the time I never form a hypothesis at all, because the answer is sitting in the 11 lines that are left.

A bug you can't reproduce is a rumor. You can't test against it, and you can't tell whether you've fixed it.

One hypothesis, and change one thing

Then, and only then, guess. And make the guess falsifiable.

“I think it’s the cache” is not a hypothesis you can act on. “If it’s the cache, then it’ll still happen with caching disabled” is, because it makes a prediction that can come out wrong.

And change one thing at a time. This is the rule everybody knows and nobody follows at hour two, and the cost of breaking it is specific: if you change three things and it works, you’ve fixed the bug and learned nothing, and you’re carrying two changes you don’t understand into the codebase.

“That can’t be it”

The most expensive sentence in this job.

You have a model of how the system works. The evidence contradicts it. And because the model has been reliable for two years and the evidence is four minutes old, you dismiss the evidence.

Every long debugging session I’ve had, and I mean every one, has had a moment somewhere in the middle where I had the answer in front of me and rejected it because it contradicted something I was certain about. The value was null and I “knew” it couldn’t be null, so I assumed the log was stale. The file was being served from somewhere else, and I knew it wasn’t, so I didn’t check.

The discipline is: when the evidence and your model disagree, the model is wrong. Always. It’s not a 50-50. The system is doing what it’s doing and your understanding is a story about it.

I’ve started saying “that can’t be it” out loud deliberately, as a flag, because if I notice myself saying it then that’s the thing to go and check first.

“Works on my machine” is a finding

This gets used as a punchline and it’s actually one of the most useful pieces of information you can get.

If it works on your machine and not on theirs, then the bug is in the difference between the machines. You’ve just eliminated everything the two have in common, which is most of the system. That’s an enormous narrowing and the correct response is enthusiasm rather than embarrassment.

So enumerate the differences: browser and version, operating system, locale and timezone, screen size, extensions, network conditions, the data in their account, whether they’re logged in, whether they’ve been logged in for a long time, what’s in their cache, whether they came from a link with query parameters on it.

Nine times out of ten it’s in that list. Locale and timezone are the two everybody forgets, and between them they’ve accounted for maybe a fifth of the genuinely baffling bugs I’ve had.

The frustration is the enemy

The part that isn’t technical.

The method above only works while you’re calm enough to run it. Past a certain level of frustration, people stop forming hypotheses and start making changes, and the changes get less justifiable as the frustration rises, and eventually you’re reverting things at random and hoping.

The tell, and it’s a reliable one: you’re making a change you can’t explain the reasoning for. The moment I catch that, I’ve learned to stop, because everything from that point is negative progress and I’m also degrading the code.

A break genuinely works, and I don’t think it’s mystical. Fixating on a wrong model is the main failure mode, and stepping away is the only reliable way I know to let go of one. The shower thing is real and it’s not inspiration, it’s the release of a grip.

And rubber-ducking works for the same reason: explaining forces you to say your assumptions out loud in order, and one of them is wrong, and you hear it.

Not much of a curriculum

The reason I wanted to write this down is that all of the above is transmitted by sitting next to somebody. It isn’t in courses, it isn’t in documentation, and it mostly isn’t in blog posts, which cover the tools and the incidents and not the method.

Which was fine when we all sat in rooms together. It’s a genuine problem now, and I don’t think enough of us have noticed that the main way this gets taught quietly stopped happening in 2020.

So: reproduce it, shrink it, one hypothesis at a time, believe the evidence, and stop when you catch yourself guessing. That’s the whole thing. It’s not much of a curriculum and it’d have saved me about four years.