200 things at once

Somebody I worked with wanted to do a redesign as a single drop. Four months on a branch, then a Monday morning where the whole site changes at once and we get to call it a relaunch. And I want to be fair to the instinct, because it’s a real one and I have it too. There’s something deeply satisfying about the clean sweep that shipping the nav one week and the type scale three weeks later just does not give you.

It went out. Conversion went down about 8%. And then everybody sat in a room for two weeks arguing about which of the roughly 200 changes had done it, which nobody could answer, because we’d run a single experiment with 200 variables in it. That’s not an experiment. That’s a vibe.

You can’t tell what did it

This is the part of big-bang releases that I think gets undersold, and it’s not the risk. Everybody already knows the risk. It’s that you’ve destroyed your own ability to learn anything from the outcome.

Ship one thing and the number moves, you know what moved it. Ship 200 things and the number moves and you know nothing at all, in either direction. A good outcome is just as uninformative as a bad one, which means the wins teach you as little as the losses. You’ve done four months of work and come out the other side with no more understanding of your users than you went in with.

And you can’t revert, either, not really. Reverting means undoing four months, so what actually happens is you go forward. You patch. You spend the next six weeks doing archaeology on your own release, and the fix for a big bang is almost always another, smaller big bang.

The platform already settled this argument

Here’s the thing that I keep coming back to. We build on top of the single most successful incrementalist project in the history of software, and we’ve mostly forgotten it was ever in question.

In the early 2000s the W3C’s plan for the future of HTML was XHTML 2.0, and it was a good design. Well-formed XML, clean semantics, draconian error handling so bad markup just fails instead of guessing. It was also explicitly not backward compatible, which is to say the entire existing web was not invited. A handful of people went off and started WHATWG on the opposite premise: extend what’s already there, pave the cowpaths, and never break a page that works. That became HTML5. XHTML 2 was abandoned in 2009 without ever shipping.

You can watch the same movie in a few other theaters. Python 3 came out in 2008 and Python 2 didn’t die until 2020, and almost all of that 12 years of pain was because it was a break rather than a path. Perl 6 was announced in 2000, arrived in 2015, and eventually got renamed to Raku because everyone had quietly agreed it was a different language. AngularJS to Angular 2 shipped with no upgrade story and split its own community in half.

Meanwhile the browsers won’t even remove a feature you’d think nobody uses until the usage counters say it’s under a fraction of a percent of page loads, and then they’ll still do an intent-to-deprecate, a deprecation warning, an origin trial for the people who need more time, and a removal several releases later. It looks like bureaucracy. It’s actually the reason a page from 1998 still renders. The web is the largest deployed system humans have ever built and it got there by never once shipping a version two.

The web is the largest deployed system humans have ever built and it got there by never once shipping a version two.

Where it’s weakest

The strongest pushback isn’t that incrementalism is slow. It’s that it’s a local maximum, and that’s a real thing, not a rhetorical concession.

You can’t get to a taller hill by only ever stepping uphill. Some changes genuinely are discontinuous: you can’t refactor your way from a jQuery soup to a component model one line at a time, and there are architectures where the small safe step in every direction is worse, so you sit on your little hill forever congratulating yourself on stability. And the related failure is drift, where every individual change was locally reasonable and the sum of them is a codebase nobody would have designed on purpose. Incrementalism with no destination is just entropy with a changelog.

Fair. I’d say two things back.

First, and this is the actual argument, the destination can be as radical as you want. It’s the delivery that shouldn’t be. Strangler fig is the standard name for it: stand the new thing up beside the old one, put a router in front, move one route across, then another, run both for as long as it takes. New writes go to the new system, backfill the old ones, delete the branch when the traffic’s at zero. That is the same rewrite. Same endpoint, same ambition. The only difference is you get to stop in the middle, and you will want to stop in the middle, and a big bang doesn’t have one.

Second, base rates. Least fun argument, and the one I trust most. Think about how many all-at-once rewrites you’ve personally watched land on schedule and deliver what the pitch promised. Now count the ones that became the second system everybody maintains alongside the first. Joel Spolsky wrote the thing about Netscape rewriting from scratch 20-odd years ago and the industry has been re-learning it roughly once per framework cycle ever since.

What you’re actually buying

Reversibility, mostly. Not caution.

You are going to be wrong. That isn’t pessimism, it’s just the base rate on decisions made with incomplete information, which is all of them. So the useful question was never “is this change correct,” it’s “how expensive is it to find out it wasn’t.” Small changes answer that cheaply. A flag you can flip at 11 pm from your phone, a canary at 5%, a dark launch that runs the new path and throws away the result while you diff it against the old one. None of that is timid. It’s just refusing to be wrong about 200 things simultaneously.

There’s a human version of it too: people mostly don’t want their tools to change on them all at once. Nobody has ever woken up hoping for a new dashboard. Every product that redesigns overnight gets the same wave of furious users, and half the time the new thing really is better, and it doesn’t matter, because you spent goodwill you didn’t have to spend. The same change rolled out over two months with a “try the new view” toggle costs you almost none of that.

This isn’t really a web development opinion

It’s just where I have to defend it most often.

I’m pretty consistently on the reform side of the reform versus revolution thing, for what I think are the same reasons. Sudden total change concentrates the risk of being wrong into one moment and one set of hands, and historically the people holding the pen when everything gets rewritten at once are not the people you’d have chosen. Whereas the boring version, the pilot program, one city, one state, the thing you can measure and repeal, gets you information and an exit. Laboratories of democracy is a slightly grandiose phrase for “try it somewhere small first,” but that is all it means. (And yes, I know the counter: some things are on fire right now and gradualism has been the polite name for never. Sometimes that’s true. It’s true less often than it’s claimed.)

And on a much smaller scale, every version of me that was going to change everything starting Monday changed nothing at all, and the things that did stick were embarrassingly small and repeated for a boring number of days. I don’t think that’s a coincidence and I don’t think it’s a different principle. It’s the same one. The compounding is the whole mechanism, and the big dramatic gesture is mostly just a way of feeling like you’ve already done it 🙃

I work almost entirely on things that are already in production and already have users, which is exactly the situation where this bias pays off. If you’re pre-launch with nobody depending on you then honestly go rewrite it, blow it up, you have nothing to lose and I’m not talking about you. But if there are people on the other end of the thing, small and reversible beats big and correct more often than it has any right to.