git bisect
Binary search over history. The command is the easy half of it.
A client mentioned, in passing, at the end of a call about something else, that the product search on their store had “been a bit odd for a while.” They thought maybe since the summer.
I checked, and yes, results for two-word queries were coming back in the wrong order. And it had been fine in June, and there were somewhere north of 900 commits between June and September, most of them mine, none of them touching anything I’d expect to affect search ranking.
My instinct, which I’ve had every time and which is wrong every time, was to sit and think about what could possibly cause that. Read some code. Form a theory. Go and check the theory.
The better move is to stop thinking entirely and let the computer narrow it down, because the computer is much better at this than I am and it doesn’t have theories.
Ten questions instead of nine hundred
git bisect is a binary search over your history. You tell it a commit where the bug exists, and a commit where it doesn’t, and it checks out the midpoint and asks you which side that one falls on. Then it halves the remaining range, and again.
900 commits is ten questions. 1,000 is ten. 100,000 is 17. That’s the whole pitch and it’s a very good pitch.
Nine hundred commits is ten questions. The computer doesn't have theories about your code, which is precisely why it's better at this than you are.
The commands are unremarkable:
git bisect start
git bisect bad # current HEAD is broken
git bisect good v2.3.0 # this tag, or a date, or a SHA, was fine
# git checks out a commit in the middle. you test it, then:
git bisect good # or: git bisect bad
# ...repeat about ten times...
# Bisecting: 0 revisions left to test after this
# a4f91c2 is the first bad commit
git bisect reset # back to where you wereIf you don’t know a good commit, git log --before="2018-06-01" -1 will give you a starting guess, and being wrong about it costs you one round.
The command is the easy half
Here’s the part that isn’t in the tutorials, and it’s the part that actually determines whether this works for you.
The hard problem is answering “is the bug present in this checkout” quickly, consistently, and without judgment. Ten times.
If your check is “start the server, log in, type two words into the search box, look at the order of the results, decide” then that’s maybe four minutes each and a decision you’ll get slightly wrong at least once around round seven when you’re bored. And getting it wrong once sends the whole search down the wrong half and gives you a confident, incorrect answer.
So most of the value here comes from turning the symptom into something with an exit code. That’s a skill and it’s transferable to every other kind of debugging, and I’d say it’s the actual subject of this post.
For the search thing it looked like this:
#!/bin/sh
npm ci --silent || exit 125 # can't build this one, skip it
npm run build --silent || exit 125
# does the second result come back before the first?
node -e '
const r = require("./dist/search-index.json");
process.exit(r.rank("blue widget")[0] === "blue-widget" ? 0 : 1);
'Exit 0 means good, anything from 1 to 124 means bad, and 125 specifically means “can’t tell, skip this one.” That last one matters more than you’d think.
Then it runs itself
Once you have that, you hand the script over and go away:
git bisect start HEAD v2.3.0 # bad first, then good
git bisect run ./check.shMine took about 11 minutes, most of which was reinstalling dependencies at each step, and I was not in the room for any of it. It printed a commit. The commit was a dependency bump, of a library I’d never looked at, which had changed a default about how it weighted terms.
I would not have found that by thinking about it. I’d have read the search code about four times and concluded that the search code was fine, which it was.
What breaks it
Dependencies that don’t match the checkout. This is the big one and it silently gives you a wrong answer. Git rolls back your source, it does not roll back node_modules or a vendor directory, so you’re testing old code against new dependencies. Which, as above, is sometimes exactly where the bug is. Install inside the test script, every time, even though it’s slow.
Commits that don’t build. Every history has them. Exit 125 and bisect will step around them and keep going, and it’ll tell you at the end if the result is ambiguous because of skips.
Enormous commits. If the answer comes back as one commit with 4,000 changed lines called “merge feature branch,” you have narrowed 900 commits down to one commit and learned very little. This is where it stops being a git problem.
It’s an argument about how you commit
The thing I took away from this, more than the command, is that bisect works exactly as well as your history is granular.
Small commits that each build, each doing one thing, mean the answer is a five-line diff and you’re done. Big end-of-week commits mean the answer is a haystack with a smaller boundary. And a history full of commits that don’t build means half your rounds are skips.
Which is a completely different reason to care about commit hygiene than the one I grew up with. I always heard it as being about tidiness, or about being considerate to reviewers, and I found both of those quite easy to ignore at 5:30 on a Friday. “Your future self can binary-search this” is a self-interested argument and I find those stick better.
Best return on effort in git
I’d used this maybe twice before, badly, both times manually, both times giving up around round six. The thing that made it click was writing the check as a script, and once I’d done that the whole shape of the tool changed from “a fiddly thing I half know” to “the first thing I reach for when something used to work.”
Ten questions. It’s the best return on effort of anything I know in git, and it finds the answer while you’re at the kettle, which is the correct place to be while a computer does a binary search.