It wrote my CSS and I rewrote it
It's very good at one particular thing, and it writes CSS like it's 2016.
I’ve been avoiding writing about this because everything published on it so far is either “this changes everything” or “it’s a plagiarism machine,” and I don’t hold either position with enough confidence to write a thousand words about it.
But I’ve now used one of these things every working day since about February, on actual client work, and I’ve got specific observations rather than a stance, so here’s the specific version.
The test that produced this post: I gave it the card component from a live project, described the layout, asked for the CSS. Nine seconds. Plausible, well-organized, commented, and about two thirds wrong.
What it was genuinely good at
Fair first, because the useful bits are real.
Anything where I know exactly what I want and simply don’t want to type it. A keyframes block. A set of @font-face declarations for four weights. Converting a designer’s list of 40 tokens into custom properties. A regex, which I get wrong about half the time unaided and which it gets right about the same proportion but much faster, and I can test it either way.
Explaining an unfamiliar error message. This one has changed my day: paste a stack trace from a tool I’ve never used and get a plausible reading of what it means, which is a much better starting point than a search that returns six threads from 2014.
And it’s a decent rubber duck. Describing a problem to it in enough detail to get a useful answer is the same act that solves about a third of problems anyway.
It writes 2016 CSS
Here’s the failure I found most interesting, because it isn’t random.
The layout it gave me used a percentage padding-top hack for the image ratio, a clearfix, a -webkit- prefix on transition, and a float-based fallback nobody has needed for years. When I asked for a responsive card it reached for breakpoints in a way I’d have written in about 2017.
That’s not a mistake so much as a bias, and the mechanism is obvious once you see it. These things are built from an enormous volume of text, and the volume is historic. There are ten years of tutorials, Stack Overflow answers and blog posts about the padding-top hack, and about 18 months of material about aspect-ratio. So the weight of evidence, statistically, is for the old way. It will keep being for the old way for years after the old way stops being right.
It's not out of date because it's behind. It's out of date because most of what has ever been written about CSS was written before the good version existed.
Which means it’s systematically weakest on exactly the things I most want help with, which are the recent ones I haven’t internalized yet. On the things I already know cold, it’s fine and I don’t need it.
I don’t think that’s fixable by a newer model, incidentally, or not entirely. It’s a property of what’s been written down.
The other four, specifically
Confidently wrong about support. It told me a feature was “widely supported” when it’s in two engines. Not hedged, not qualified. This is the class that worries me most, because browser support is precisely the sort of fact you’d want to look up, and it produces an answer with exactly the same tone as its correct ones.
Selectors that read correctly and don’t work. A :not() with two arguments used in a way that inverted the meaning, and a specificity assumption that was wrong. Both looked completely reasonable. Both would have been caught in the browser in about 30 seconds, which is the saving grace of CSS: the feedback loop is short and visual.
The accessibility defaults are poor. Unprompted, I got a clickable div, outline: none on a focus state, placeholder text doing the job of a label, and an error state indicated by color alone. Every one of those is the majority pattern on the web, so of course it’s the majority pattern in the output. It’s a mirror, and the average of the web is not good, and if you don’t already know that list you’ll ship it.
It doesn’t know my codebase. It invented class names in a convention we don’t use, duplicated a component we already have three files away, and reached for a library for something our project does natively. Which is unfair as a criticism, it had no way to know. It’s just that “produces code that fits this project” is most of what makes code good, and that’s the part it can’t do.
Fluency reads as competence
The thing I’m most wary of, and it’s not really about the technology.
It is extraordinarily fluent. The prose is confident, the code is tidy, the comments are apt, and the whole thing has the texture of something written by somebody who knows what they’re doing.
And I evaluate text by fluency. Everybody does. It’s how we’ve assessed competence in writing for our entire lives, because until about a year ago fluency and understanding were correlated, since producing fluent text about a subject required knowing the subject.
That correlation has broken and my instincts haven’t updated. I caught myself accepting the confident support claim above, briefly, purely on tone, and I have 15 years of reasons to check that particular kind of claim.
Which is why I’m most uneasy about this for people early in their career, and it’s not a “kids these days” thing. I could tell the CSS was from 2016 because I wrote CSS in 2016. Somebody two years in has no way to distinguish confident-and-current from confident-and-obsolete, and the output gives them no signal at all.
Where I’ve kept it
Rubber-ducking. Naming things, which I once wrote a whole post about being genuinely hard, and for which “give me 20 options” is a legitimately good use of a machine that has read everything. I don’t take its suggestions. I react to them, and reacting is easier than generating.
First drafts of tedious configuration.
Explaining unfamiliar errors and unfamiliar code, with the caveat that I verify anything I’m going to act on.
And, honestly, the rewriting-a-thing-that-exists effect. Even when the output was two thirds wrong, having two thirds wrong in front of me was a faster route to the right answer than an empty file. Editing is easier than writing. That’s a real benefit and it survives the output being poor, which I find slightly odd and can’t argue with.
Where I’ve drawn the line
The rule I’ve ended up with, which I’d offer as the one useful sentence here: I’ll accept code I couldn’t be bothered to write, and I won’t accept code I couldn’t have written.
That draws the line in the right place, I think. It keeps all the speed on the tedious things, and it means nothing lands in a codebase I maintain that I can’t explain, debug, or take responsibility for at 4:30 on a Friday.
Ask me again in a year. I’ve written enough posts about tools to know how this goes 🙃