Off my laptop
Deploying from a train station, and the Tuesday a pipeline caught something I'd have shipped.
Two years ago I deployed a client site from a departure gate at Lindbergh, on a phone hotspot, standing up, because something needed to go live before a meeting and I was the only person who could do it.
It worked. It nearly always worked. That’s the problem with describing this as a disaster story: there wasn’t a disaster, there was five years of it being fine, which is exactly the condition under which nobody changes anything.
What eventually got me was a much smaller thing. A colleague pulled a project I’d been on, ran the build, and got a different result than I did. Not an error, a different result, and it took us most of an afternoon to establish that the version of a tool installed globally on my machine in 2016 was doing something the version in the project’s manifest wasn’t.
Which means the site that had been live for eight months was not built from that repository. It was built from that repository plus the accumulated sediment of my laptop, and nobody could reproduce it, including me, if I’d got a new machine.
What was actually wrong with it
Not the FTP, though it was FTP for the first couple of years and then rsync and then a push to a hook on the server, and each of those was an improvement.
The problem is that “deploy” was a thing that happened to my machine’s current state, whatever that was. Which meant the answer to “what’s on production” was never “the contents of master.” It was “whatever was in my working directory at 4:50 pm on the 14th,” and those two things are the same on most days and not on the days that matter.
The answer to "what's on production" was never "the contents of master." It was "whatever was in my working directory at 4:50."
Specific ways that went wrong, all of which I have done:
Deployed with uncommitted changes, so the fix exists on the server and nowhere else, and it survives until the next deploy overwrites it and the bug comes back looking like a haunting.
Deployed the wrong branch, because I’d been on the feature branch that morning.
Built with dependencies that were only on my machine, as above.
And the structural one, which is that nobody else could do it. Not for lack of skill, for lack of the sediment. If I’d been ill on the day of a launch, that launch didn’t happen, and that’s a thing I’d said out loud in a meeting more than once as though it were a fact about the world rather than a decision I’d made by default.
The artifact comes from the repo
The change, once you strip out all the tooling talk, is one sentence: something that isn’t a person checks out a clean copy, builds it, and puts the result on the server.
Everything else falls out of that. It’s reproducible, because a clean checkout has no sediment in it. It’s auditable, because there’s a log of what was deployed and when and from which commit. It can be done by anyone with commit access, or by nobody, which is the real version. And “what’s on production” becomes a question with an answer.
For a static site the pipeline is genuinely small. Mine, roughly, with the provider details filed off because there are half a dozen of these now and they’re all much of a muchness:
steps:
- script:
- npm ci # clean install from the lockfile, not npm install
- bundle install --deployment
- bundle exec jekyll build
- npm run check # the link checker and the smoke assertions
- deploy:
branch: master # nothing else deploys, ever
script:
- rsync -az --delete _site/ $HOST:$PATHnpm ci rather than npm install matters more than it looks: it installs exactly what the lockfile says and fails if the lockfile and the manifest disagree, which is the whole point of doing this away from my machine. Using install here would quietly reintroduce the drift I was trying to remove.
The Tuesday it earned its keep
Six weeks in.
The client’s marketing team had restructured a section in the CMS, renaming three pages, which they were entitled to do and which nobody thought to mention. The next deploy ran the link check, found 11 internal links pointing at pages that no longer existed, and refused.
I’d have shipped that. I’d have shipped it with complete confidence, because the site looked perfect on my machine, because on my machine I’d never clicked those links either. It would have sat there until somebody complained, and the somebody would have been a customer rather than us.
That was the moment the whole thing stopped being a hygiene project and started being obviously worth it. Not because CI is clever, but because there is now a thing that is not me, that does not get tired, that checks the same list every single time.
What it costs
It isn’t free and I’d rather say so.
Build minutes cost money on anything but the smallest plans, and a pipeline that takes six minutes on a site you edit daily adds up. Slow pipelines also change behavior badly: if a deploy takes ten minutes, people stop deploying, and you’re back to batching.
Secrets have to live somewhere. They were on my laptop, which was bad but at least comprehensible. Now they’re in a provider’s settings panel, which is better, and it’s a thing to think about rather than a thing that’s gone.
And you’ve added a dependency on somebody else being up. I had a launch delayed 40 minutes last fall because a status page had gone yellow, and standing there unable to deploy a static site to a server I have SSH access to felt genuinely stupid. I now keep the manual command in the README for exactly that, and I’ve used it once.
Where I’ve drawn the line
I’m not evangelical about this and about half my projects still deploy from my machine. For a site that changes twice a year and is owned by a client who’ll be gone in two, setting up a pipeline is ceremony, and I’d rather spend that hour on the print stylesheet.
The line I’ve settled on is: if more than one person touches it, or if it changes more than monthly, or if it takes money, it comes out of a pipeline. Otherwise it comes out of me, and the README says so.
But I do think about the eight months where the live site wasn’t reproducible from its own repository, and how completely unaware of that I was, and how it wasn’t a mistake anybody made. It was just the shape of the thing.