We billed them for the cleanup

A site I built in 2018, at the agency I was at before this one, spent about two months redirecting phones somewhere else. Desktop was fine. Everybody who touched that site, on either side, checked it on a laptop, which is most of why it was two months and not two days.

We found out because a customer emailed the client to ask why their site kept sending them to a page about a sweepstakes. Not from monitoring, which there wasn’t any of. Not from us.

terminal

$ find wp-content/uploads -name '*.php' -printf '%TY-%Tm-%Td  %p\n'
2021-02-17  wp-content/uploads/2018/09/wp-conf.php
2021-02-17  wp-content/uploads/2019/04/class-wp-cache.php
2021-02-17  wp-content/uploads/2020/11/index.php

Shell

Three PHP files in the uploads folder, all written the same night in February. Nothing in that folder is ever supposed to end in .php. It holds the pictures.

The way in was a plugin about two years behind, with a hole that had been published and patched long before any of this, found by something automated that reads the entire internet looking for exactly that plugin at exactly that version. It has had its own entry on the OWASP list for as long as there has been a list, under the least dramatic name on there. Nobody picked that site. It answered a knock.

We cleaned it up over a couple of days, and then we invoiced for the couple of days, and that invoice was comfortably more than a year of the maintenance nobody had ever offered them. I don’t think a single person in the building found that strange (me very much included!)

And nobody did anything wrong at any point, which is the part I keep coming back to a year later. We built what was scoped. They paid what was quoted. Nobody bought maintenance because nobody sold maintenance, and the reason nobody sold it isn’t negligence, it’s the shape of the thing. Same shape as a homepage that grows to 2.4 MB because people changed the picture like they were supposed to be able to. Everyone behaves correctly and the outcome is bad anyway.

Nothing to photograph

From our side the incentives all point one direction. I say that as somebody who benefits from them.

A build is a project. It has a start, an end, a scope, a fixed price and a screenshot for the website. Maintenance has none of that. It’s open ended, the effort moves around month to month. There is nothing to photograph at the end of it.

New business is also what gets celebrated. I’ve sat in plenty of weekly meetings where somebody won a build and the room clapped. I have never once heard anyone announce that a retainer had renewed for the sixth year.

But the real difficulty is that the product is a non-event. You’re selling that nothing bad happens, so when it’s working, the client’s experience of it is identical to not having bought it. After 18 months of nothing happening the line item starts to look like a streaming service somebody forgot to cancel.

Like the office carpet

The client side is just as reasonable, which is the part that makes it hard to argue with.

The site is done. It launched, everyone was pleased, and now it’s a thing that exists, like the office carpet. Nobody has a mental model where the carpet has a version number and a support window. And nothing about how we sell these, deliver them or hand them over does anything to correct that.

The money is in the wrong shape too. A build is one amount, approved once, usually out of a budget that exists for projects. Ongoing monthly spend goes through a completely different door, gets reviewed every year by somebody who wasn’t in the original room, and has to survive next to line items with visible outputs.

And the word itself has been spent. Nearly every client I’ve talked to about this has previously paid somebody a couple hundred a month and gotten back an automatically generated PDF listing plugin updates plus a traffic chart that all reads as very deliverable-coded. So “maintenance” arrives already meaning money for nothing, and I can’t really argue because I’ve read those reports.

Everything with a date on it

Worth being concrete here, because “things decay” has never persuaded anybody of anything.

Security updates, which is the one everyone names and the one that got that 2018 site.

The platform underneath. The language runtime has a support end date, published years in advance. When it passes, the host moves the site onto a newer version on their schedule, sometimes with a couple weeks’ notice, and if the code isn’t ready the site breaks. It’s the most predictable failure on this list and the one that surprises clients most, because it arrives from a direction they didn’t know was there.

Integrations. The payment provider deprecates an API version. The mailing platform moves an endpoint. The map service starts requiring a billing account. Every one of those is somebody else’s roadmap arriving on a day you didn’t pick.

The web moving underneath it. Cookie rules tightened, SameSite defaults changed, browsers started blocking things they used to allow. A site that worked at launch stops doing one specific thing, and nobody notices for months, same as the phones. The other half of that is the code nobody takes back out, and a browser going out of support last month turned up a polyfill in a client template that had been doing nothing since about 2016.

Certificates, domains and DNS, which is a whole post from 2017 and still causes more emergencies than code does.

Packages nobody chose, four levels down a dependency tree nobody reads. They have not gotten better since.

And the content. Dead links, a team page listing three people who left, a notice about opening hours from 2020. Nobody files any of that under maintenance and it’s the part customers actually see.

Write down the nothing

Of everything here, this is the one I’d actually do first.

Retainers get canceled because nothing visible happens, so make the nothing visible. A short note every month, in plain words, saying what was done and what didn’t happen as a result of it.

the monthly note

June

- Applied 14 updates. Three were security fixes, and one of those
  was for a hole being actively exploited on other sites right now.
- Restored May's backup into a test environment and it came back
  clean. We do this quarterly, because a backup nobody has restored
  is a rumor.
- Submitted the contact form and the quote form. Both arrived.
- Checked the certificate. It renews automatically on October 12.
- The version of PHP this site runs on stops getting security
  patches in about 14 months, so it's worth planning that upgrade
  for next spring instead of doing it in a hurry.

Text

That takes about 15 minutes to write and I think it’s most of the difference between renewing and not renewing. I’d guess it’s the highest-return quarter of an hour available anywhere in this line of work. It isn’t close.

The last line in there is doing something extra, too. A client told 14 months ahead of a platform upgrade experiences you as somebody looking after them. The same client, told with three weeks’ notice, experiences an emergency invoice. It’s the same work either way.

What I’d want in the proposal

I don’t write the proposals. When I’ve been asked, though, this is roughly what I’ve argued for, and the first part is just saying a number out loud with a reason attached to it.

The rough convention across software is somewhere around 15 to 20% of the build cost per year for ongoing support. Saying that helps, because it’s a figure the client can go and check instead of a number that came from us.

Then two things that aren’t strictly maintenance. The first is a small pot of hours every month for small changes, spendable on whatever they want. That matters more than it sounds, because it turns a grudge purchase into something they actually use, so the relationship stays warm, so when something does go wrong you’re already somebody they talk to instead of somebody they have to call. The second is an annual written review of what’s coming, what’s aging and what to budget for. One page, and the only document in the whole arrangement that’s clearly worth money by itself.

I knew about the PHP version

I’m not the one who sells any of this. Retainers get sold by people whose job is selling, and for most of my career I’ve been quietly glad about that in a way I’ve never examined very hard.

Because I am the one who knows. I can read a lockfile. I can see that the runtime has 14 months left on it, that a plugin hasn’t been touched since 2019, that the certificate is manual and the person who used to renew it left in the spring. None of that is a mystery. It’s half an hour and a terminal. And I have mostly written it into a ticket, or said it once on a call, and then let it go, and called that staying in my lane.

Which is a very comfortable thing to believe when the alternative is a difficult conversation with somebody’s finance department in it. I’ve also noticed that the way I describe that conversation to myself, that it feels like selling fear, does a lot of convenient work for me. It lets me skip the hard part and file the skipping under integrity.

It isn’t fear, though. Fear is “you’ll get hacked.” The true version is that the PHP this runs on stops getting security patches in 14 months, here is what that costs to do in the spring, and here is what it costs in a panic in December. Nobody has to be frightened for that to be worth hearing. And I’ve now seen the other version up close, where nobody says it, and the work happens anyway about a year late at a much worse price. We send the invoice, and it gets paid without a murmur, because by then there is no decision left in it for anyone to make.