The tests I actually write

A site I’d worked on went live with <meta name="robots" content="noindex"> still on it. It had been on staging for the obvious reason and the build didn’t strip it, because the build had never been told to.

Eleven days. Nobody noticed until organic traffic showed up as a flat line in a monthly report, and by then the client had also spent money driving people to a page that was fine, so it wasn’t catastrophic, it was just embarrassing and expensive in the way that’s hardest to explain.

Nothing in any testing methodology I’d read at the time would have caught that. There was no unit to test. It was one attribute, in one layout file, that was correct in one environment and wrong in another, and the only way to know is to look at the actual thing on the actual server.

That’s roughly when I stopped trying to have a test suite and started trying to have a list of the ways this actually goes wrong.

The pyramid assumes there’s logic

The standard advice is a broad base of unit tests, fewer integration tests, a handful of end-to-end ones. It’s good advice for software that computes things. A pricing engine, a permissions model, a date library.

A marketing site computes almost nothing, and I want to be clear this post is about those rather than about the stores, which are all logic and get tested very differently. A brochure build is a set of templates, some content from a CMS, a stylesheet, and maybe 200 lines of JavaScript for a menu and a form. When I’ve tried to write unit tests for that, I’ve found myself extracting a function specifically so there’d be something with a return value in it, and then testing that function, and the test passes forever because that function was never going to break.

I found myself extracting a function specifically so there'd be something with a return value to test, then testing it, and it passed forever.

Meanwhile the things that do break are: a link, a build, a form, and a piece of configuration that differs between environments. None of which is a unit.

Four piles of broken

I went back through two years of “the site is broken” emails to see what they actually were, which I’d recommend to anyone as a Sunday afternoon exercise. Mine sorted into four piles.

Links, by a mile. Someone renames a page in the CMS, or a section gets restructured, and 11 internal links now 404. Nobody finds out because nobody clicks their own footer.

Environment configuration. The noindex thing. An analytics snippet that’s on staging and not production, or vice versa. A form posting to a test endpoint. A Disallow: / in robots.txt. Each of these is invisible in the browser and each of them silently disables the entire commercial point of the site.

The one interactive thing. Every brochure site has exactly one path that carries money or leads, and it’s the contact form. When it breaks, it breaks silently, because a form that posts into nothing looks exactly like a form that worked.

And the build. Which sounds trivial and is about a third of everything, because half of these projects sit untouched for eight months and then a dependency has moved, and “does this still build from a clean clone” is a real question with a real failure rate.

The four I write

So, in order of value per minute spent.

Does it build, from clean, on something that isn’t my laptop. This is the cheapest and catches the most. It is also, embarrassingly, a test.

A link check over the built output. For a static build this is one command over the whole _site directory and it catches renames, typos, and the trailing-slash thing that only breaks on the live server.

Rakefile

task :check do
  sh "bundle exec jekyll build"
  HTMLProofer.check_directory("./_site", {
    :check_html      => true,
    :check_img_http  => true,   # mixed content, which bites after an SSL move
    :url_ignore      => [/linkedin\.com/]  # they 999 every bot, it's not you
  }).run
end

Ruby

Assertions about the deployed page, not the built file. This is the noindex lesson and it’s the one I’d fight for. A short script that fetches the real production URL and asserts a handful of things that must be true in that environment and only that environment.

smoke.js

const html = await fetch('https://example.com/').then(r => r.text());

assert(!html.includes('noindex'), 'production is asking not to be indexed');
assert(html.includes('UA-'), 'analytics snippet is missing');
assert(html.includes('<title>'), 'no title tag');
assert((await fetch('https://example.com/robots.txt').then(r => r.text()))
  .indexOf('Disallow: /') === -1, 'robots.txt is blocking everything');

JavaScript

Four lines, and every one of them is a real outage I’ve either caused or inherited.

One end-to-end test of the form. Fill it in, submit it, assert the success state, and if you can, assert something received it. The browser-driving tools got dramatically less painful in the last year or so and this is now maybe 40 minutes of work rather than a day of fighting a driver.

Visual regression, twice

Visual regression testing, twice now, and I want to be fair to it because when it works it’s remarkable.

The problem is that on a real project the diff is never empty. Fonts render half a pixel differently between runs, a CMS-driven testimonial changes, an embedded map tile loads differently, and the ad script that the client’s marketing team added last week shifts everything down 40px on Tuesdays. So every run has 30 differences in it and 29 are noise, and within about three weeks everyone approves the diffs without looking, which is worse than not having it, because now you believe you’re covered.

I’d use it on a design system page with fixed content, where the inputs genuinely don’t move. On a live marketing site I’ve never got the signal-to-noise anywhere near good enough, and I’d be glad to be told I’m doing it wrong.

The checklist is a test

The last thing, and I’ve stopped being sheepish about it: most of what protects a small site is a human going through a list before launch.

Print preview. Tab through the form. Turn images off. Check it in the second browser. Look at the 404 page. Send yourself the confirmation email and read it on a phone.

That’s a test suite. It’s executed on wetware and it doesn’t run on every commit, and for a site that changes four times a year that’s a completely defensible trade-off. Writing the list down is 90% of the value, because the failure isn’t that people skip checks, it’s that they forget which checks exist, and it’s a different person each time.

One rule

A test earns its place by corresponding to a real email I’ve received. Not a category of bug, an actual message from an actual client saying something is wrong.

By that standard I have four automated tests and a checklist on most projects, and I don’t feel bad about it any more. The suite that would have caught the noindex thing has one assertion in it and would have taken four minutes to write, and I’d have swapped it for every unit test I’ve ever written for a website.