A review of my own change told me something yesterday that I did not want to hear. Nothing in the test suite required the pagination links on this site to exist. Blank out the function that renders them, regenerate the pages, run everything: 178 passed. The second page of the blog was still generated, still listed in the sitemap, and reachable from nowhere a person could click — which is the exact condition the pagination had been built to remove.
The suite was not broken. It ran, it read the pages, it made assertions, and every one of them was true. It simply never asked the question the feature existed to answer.
What it did check was real, and that is what makes the case worth telling. It walked the list of pages by following the rel=next hint that each page carries in its head — a machine-readable pointer saying which page comes after this one. Every page in the chain was reached, every one was accounted for, nothing was missing. The walk was correct.
But no browser renders that hint and no visitor follows it. On a phone it is invisible. The thing a person needs is a link under the articles that says Older, and the suite had no opinion about whether that link existed. It had tested the instructions to the machine and not the door for the human, and the two had been written by the same hand on the same afternoon, so they agreed with each other perfectly.
The repair was one loop. The walk was already collecting every link it saw on each page as it went; it simply never compared that collection to the pages it had walked. Requiring every page in the chain to also appear among the links a visitor could click turns the same walk into the test that was missing.
A failing check reports itself. A passing one does not
This is the asymmetry the whole trade is built on and almost nobody states. A check that goes red tells you about itself: something is wrong, come and look. A check that stays green makes a claim — the thing I watch is fine — and it makes that claim in exactly the same words whether it is watching or not. Green is indistinguishable from blind, from the outside, forever.
So a green check is an assertion about the system, and like any assertion it is worth nothing until somebody shows it can fail. Not that it does fail — that it can, on the precise breakage it was posted against. Until you have seen it go red for the right reason, you have a green light and no evidence.
Five of ours, inside five weeks
I went looking for how often this had already happened to us, and did not have to look far. These five are from our own repositories, in the last five weeks, and every one of them was found by something other than the check that should have found it.
| Where | What the green meant |
|---|---|
| DRL088#2061 | Nineteen tests against a live database had never run once. The marker selecting them was one word wrong. |
| GGM016#73 | A watchdog over the deployed surface covered less than it claimed, and said nothing about the gap. |
| GGM016#75 | An end-to-end suite proved what had already been checked, measuring the wrong thing and not on a clock it controlled. |
| DRL088#2086 | Two guards reached the branch they were meant to inspect and could not see inside it. |
| GGM000#9520 | The receipt for a full run named content that the run had never read. |
The shapes do not repeat. One never ran at all. One covered less than it claimed. One measured the wrong thing. One reached the edge of what it was meant to inspect and stopped. One reported on content it had never read. Five checks, five different ways to be green — which is why the lesson cannot be write better tests. There is no single failure to guard against. None of them was written carelessly either: every one had been reviewed and merged by people who wanted it to work, and every one had been green from the day it landed.
What that costs, as a share of the repair work
Some of our repair work does not go into the product at all — it goes into the apparatus that checks the product. In August, of 1548 merged changes, 776 were repairs; of those, 66 — 8.5 per cent — repaired a check rather than the thing being checked. Over the first three days of September the same count over 116 repairs gives 16, or 13.8 per cent.
The number I would defend is the modest one: a serious fraction of everything we repair is the machinery of checking, not the thing being sold. That machinery is not overhead around the work. It is the work, and it breaks like the rest of it.
The count is a check too, and it does not see everything either
Which is worth showing, because I caught my own measurement doing the same thing today. Take the same month, the same repositories, the same question — how many of these changes were repairs — and count it twice. The first count asks whether the title starts with the word: 785. The second asks for the word, an optional scope in brackets, and then a colon: 776. Nine changes sit between the two answers.
And they are not junk. They are ordinary repairs written in a different house style — Fix/critical endpoint disable, fixed broken tests, added minio init for docker compose, fix/feat(#1396): контраст, локаль-ссылки и воронка интереса. Every one is real work that really fixed something. A stricter rule calls them nothing, a looser rule counts them, and neither rule is wrong. The number simply does not exist without the definition attached to it.
The narrower count moves further. The same three days of September give 16 repairs to the checking machinery under the word list published with this article, and an earlier pass over the identical window, with a different list, recorded 22. Same days, same repositories, same underlying work: the disagreement is entirely in what the two lists were looking for.
The rule that produced every figure above
Pull requests merged in the GodModeLab and DreamLightLabs organisations, counted by merge date in UTC. A change counts as a repair when its title matches
^fix(\([^)]*\))?!?:
and it counts as a repair to the checking machinery when the title also contains one of these, in either language:
guard, check, test, e2e, lint, assert, coverage, сторож, гейт, приёмка, приемка, квитанция, тест, проверк
The keyword match runs on the title only, so a repair to a check named after the breakage it failed to catch is not counted. Both shares are floors.
What we changed, and it is one sentence
A new test now has to be shown failing. Not described as covering something — shown: name a specific change to the source that the test should catch, make that change, watch the test go red, put it back.
In the change that came after the review, eight new tests went in, and against the rule that was there before them six of them go red. That is the whole ceremony, and it converts a sentence nobody can check — the test passes — into one anybody can: the test would have caught this.
The other two stayed green, and I am reporting that rather than the round number I first wrote down. Both of them assert that something is absent — that a page carries no date it has no business carrying. The old rule gave those pages no date at all, so both were satisfied by the very defect they were written against. A test can be green because the bug is present. That is this article arriving one layer up, inside the change that was supposed to be its answer.
And the part that keeps me honest about it
On the change the review caught, I had already done this. I raised seven deliberate breakages against my own new tests and every one of them was caught. The suite was still blind to the thing that mattered, and a reader found it in an afternoon.
The reason is worth more than the technique. I had mutated what I built — the page numbering, the boundaries, the file names — because that is what was in front of me. The failure was in what a visitor needs: a link to click. You cannot mutate the thing you did not think of, so a killed mutant proves the test binds that one change and says nothing whatever about the ones you never imagined.
The line I would keep
Green is a claim. Ask it the same thing you would ask any other claim on a status page: what would have to be true for this to be false, and have I ever seen it say so? If the honest answer is that it has been green since the day it was written, that is not a record of health. That is an untested assertion, and I have six of them from the last five weeks to show what they cost.