Cron schedules and cache windows
On a call in November about something else entirely, the merchandiser at a client asked me how long a price change usually takes to show up on the site. I said it should be right away.
They said it isn’t. They said it’s about 40 minutes if you do it in the morning and that after 4 pm it might not show up until the next day. So they do prices before lunch. During a sale they do the whole batch at once and go find something else to work on for an hour.
Every part of that was correct. I couldn’t have told you a single number in it, and neither could anybody else on our side.
Two places a price lives
When you type a price into the admin and save it, it goes into the tables shaped like the data. It gets a row holding the value and pointing at the product, the attribute, and the store. It’s correct there the instant the save finishes. The product page is correct the instant the save finishes too because it reads those tables more or less directly.
Category pages don’t read those tables. Almost nothing a customer looks at reads those tables.
A listing reads a flat table that a job builds out of the real ones in advance on a schedule. It has one row per product, and the row holds the answer instead of the ingredients. Working the answer out on demand means resolving the customer group, the tier prices, whatever catalog rule is active, and the stock position for every product in the grid. And that’s on the page that gets the most traffic in the store, for a person who’s going to scroll past most of it anyway.
So the platform does the work once, writes it down, and serves the writing. I wrote in February about putting a full-page cache in front of a store. This is the same bargain one layer down.
That one stores the finished page and this one stores the finished answer. Both are a promise to be a little out of date in exchange for being quick. I’d defend both of them.
They also stack, which I hadn’t thought through until this came up. Saving a product purges every cached page it appears on (category pages included). The next person to open one of those puts it right back, built from a flat table that hasn’t caught up yet.
So for the whole window the cache is busy re-caching the old price. It only lets go when the job finishes and the platform purges those pages a second time. Refreshing doesn’t help because the page you get back is a faithful copy of the index, and the index is behind.
Nobody had said out loud who’s on the other end of that promise.
Ready when you are
There’s a screen in the admin that shows the state of all this. I’d bet money the merchandiser had never been walked through it.
Indexer Mode Status
Product Price Update by Schedule READY
Product Flat Data Update by Schedule READY
Category Products Update by Schedule READY
Catalog Search Update by Schedule READY“Ready” isn’t a claim about your change. On a scheduled index it means that nothing has marked the whole index as needing a rebuild, which it’ll happily say with a few hundred saved products waiting in line behind it. The line itself isn’t on the screen at all.
So it says Ready at 9 am and it says Ready 30 seconds before the job runs. The one place in the whole admin that could have answered the question can’t.
The dropdown nobody remembers choosing
Set the catalog to reindex on save and the flat tables are right within seconds. That’s a delight on a catalog of 400 products. On this one it’s a disaster because every row of the nightly feed would trigger its own pass and the store would be face down for an hour every night.
Set it to reindex on a schedule and the site stays up through the import, at the cost of a window where the listings are knowingly behind.
This store takes a couple thousand rows off an overnight ERP feed so reindexing on save was never going to be the answer. Scheduled is right. It’s just that scheduled got chosen during a launch by somebody reasoning about server load and then got filed as a hosting detail. That dropdown actually decides how long a person at the client sits looking at a browser window before they believe themselves.
The afternoon was a queue
I could account for the morning number on the spot. I couldn’t account for the afternoon one. I think it’s the more interesting half.
The job doesn’t recompute the catalog from scratch every time. It works in order through whatever has been flagged since the last cron run. That’s fast when a merchandiser has touched six products and slow when something has touched a few thousand. And the feed was landing at 4:30 in the afternoon.
So a price typed at 4:40 wasn’t waiting 40 minutes. It was sitting in line behind the entire feed. By the time the line cleared, the useful part of the day was over. Not a delay, a rush hour.
Their rule about the afternoon was a real observation about a queue they’d never been told existed, and they got there by watching a page. Their notes had the rush hour in them and our documentation didn’t have the queue.
Nobody could say why it was 4:30
I asked why the feed lands in the middle of the afternoon.
Nobody could say. That includes the host, the developer who wrote the connector (who’d left), and the person at the client whose ERP it is.
As best anybody could tell, it was set up that way during a testing week before launch at a time that suited whoever was testing it. It hadn’t been touched since. That’s most of the reason anything anywhere is set to any particular time, I think.
It moved to 2 am. That took one email and then a couple weeks of waiting for the right person to be back. After that the afternoon half of the merchandiser’s rule stopped being true.
The morning half stayed true because the morning half isn’t a fault. So I did what I could with it. I said the number out loud, wrote it into a document with their name on it, and then said it again on the next two calls because a thing you say once during a handover is a thing nobody has.
Prices before lunch
Since then I’ve been chewing on where that knowledge was living.
It wasn’t in a README or a ticket or anyone’s head at the agency. It was in one person at the client. They’d worked it out entirely from the outside over about a year and a half by making a change and then refreshing a category page in a second browser window until it moved. And it was accurate to the nearest 10 minutes.
I wrote in October about how a project stops being legible to the person who built it after about eight weeks away. This is the same hole seen from the other side. The client had documentation of our system. We had a dropdown and a cron line on a server none of us had opened in a year.
They probably do the checking because they got burned once and had to explain to somebody why a price on the site wasn’t the price in the meeting. I think that’s what most workarounds are. A system declines to explain itself, and a person invents a little ritual to cover the gap. The ritual works, and then a whole team is relying on it and nobody has written it down.
They kept the second browser window after we moved the feed, by the way. I found that out in December and I think it’s entirely reasonable. A number from a developer is a claim about how a system behaves. A year and a half of watching it is evidence.
If I want the checking to stop, the right way isn’t to tell them 40 minutes. It’s to put the time of the last successful run somewhere they already have open. I haven’t done that yet.