The agent wrote a test that passes
Six ways to write a green test that verifies nothing, and the 30-second check that finds all of them.
14 tests, generated in about a minute, for a pricing module I’d been putting off covering for a year. Ran them. All green. Coverage went from about 9% to 60%.
Then I read them, which took considerably longer than generating them, and roughly half were incapable of failing.
Not wrong. Not badly written. Incapable of failing, in the sense that no possible change to the code under test would have made them go red.
Six ways to pass without testing anything
Sorted roughly by how convincing they look.
The tautology. expect(result).toBeDefined(). expect(typeof total).toBe('number'). These are true whether the calculation is right or catastrophically wrong, and they look like assertions because they’re written in assertion syntax.
Asserting the mock. The dependency is mocked to return a value, and then the test asserts that the function returned that value. What’s been verified is that the mocking library works.
A snapshot of current behavior. This is the insidious one. Run the code, capture what it produces, save it as the expected output. Now the test asserts that the code does what it currently does, which includes every bug it currently has, and it will go red the first time somebody fixes something. It looks like the most thorough test in the file.
Catching your own exception. A try with the call in it, a catch with nothing in it, and an assertion after the block. If the function throws, the catch swallows it and the test carries on and passes.
Testing the language. Asserting that a property returns the value it was just assigned. Something is being verified there and it isn’t your code.
No failing case at all. 14 happy paths. Nothing with an empty array, a null, a negative number, a boundary, a currency with no minor units, or a discount larger than the total. Which was, in this specific module, exactly where the bugs were.
Half of them weren't wrong. They were incapable of failing, which is a different and worse thing.
A test’s only value is that it can fail
This is the sentence I’d like to nail to something.
A test is not valuable because it exists, or because it’s green, or because it covers a line. It is valuable purely and entirely in proportion to its ability to go red when the behavior is wrong.
Which means a test that cannot fail has negative value. It costs runtime. It costs maintenance, because somebody will have to update it during a refactor. It occupies a line in the coverage report. And, worst, it produces confidence, and the confidence is spent on decisions.
A module with no tests is honestly untested and everybody treats it accordingly. A module with 14 decorative tests is untested and nobody knows.
Break the code on purpose
The check, and it takes about 30 seconds per test.
Go into the function and break it deliberately. Flip a comparison. Return zero. Change a plus to a minus. Delete a line. Then run the tests.
If they’re still green, the test is decorative and you can delete it.
// original
if (subtotal >= threshold) discount = subtotal * rate;
// break it, deliberately, and run the suite
if (subtotal < threshold) discount = subtotal * rate;
// ^^ if nothing goes red, nothing was testing thisThere’s a whole discipline built on this called mutation testing, with tools that do it systematically, and I’ve never got one properly set up on a client project. The hand version needs no tooling and I now do it on any test I didn’t write myself, which since about January means most of them.
Four of the 14 survived. I kept those, rewrote five, and deleted the rest.
Why it does this, which isn’t mysterious
I want to be careful not to make this spookier than it is.
An agent asked to write tests is optimizing for producing something that looks like a test file and comes back green. That’s the visible success condition: the run passes, the coverage number goes up, the task is done.
Under that objective, an assertion that cannot fail is not a mistake. It’s a perfect strategy. It satisfies every observable criterion at minimum risk. Nothing in the loop distinguishes a test that verifies behavior from a test that verifies nothing, because both are green, and green is the signal.
That’s a misspecified objective rather than a defect of understanding, and it’s the same reason a coverage target produces the same tests when you give it to a person.
Which humans do too
And I’d be a fraud if I didn’t say this.
I have written expect(result).toBeDefined() to get a coverage number over a threshold before a release. More than once. I knew exactly what I was doing and I did it at about 5:30.
Coverage is a measurement of lines executed, not of behavior verified, and the moment it becomes a target it stops measuring anything, because the cheapest way to execute a line is to call it and assert nothing. That’s not a new observation and we’ve been living with it for 20 years.
What’s changed is throughput. A person who’s decided to game a coverage number produces maybe a dozen useless tests before getting bored. There’s no longer a bottleneck on that.
What I actually let it do
The split I landed on in 2023 still holds: accept the work I couldn’t be bothered to do, not the work I couldn’t do.
So it writes the setup. The fixtures, the mock scaffolding, the parameterized table of cases, the tedious arrangement of five slightly different input objects. That’s genuinely a chore and it’s good at it.
I write the assertions. All of them. They’re the part that encodes what correct means, and encoding what correct means is the actual job.
And there’s one prompt that has turned out to be worth more than “write tests,” which is asking what input would break this. It’s very good at enumerating edge cases, because that’s a recall problem across a huge amount of prior art, and it’s poor at judging whether an assertion is meaningful, because that requires knowing what the code is for. Play to the first and don’t rely on the second.
Fourteen tests, four kept
The unnerving part isn’t that it wrote bad tests. It’s how completely convincing the file looked, and that if I’d been busier I’d have read the green tick, felt covered, and moved on. The confidence was the deliverable, and the confidence was the thing that was wrong 🫠