AI Agent Tests All Green? Why That Proves Nothing When You Compare Setups

Today I want to give you one question. Ask it every time your AI agent's tests come back green, and you will stop making a mistake that almost everyone makes. It takes ten seconds, and it can save you from shipping a decision that your tests never actually checked.

Here is the problem.

You change something in how your agent works. A different model. A new prompt. You start clearing the context more often, or less often. You run it again. And the tests come back. All green. Not a single failure.

So you conclude: nothing broke. The cheaper setup is just as good as the expensive one. You switch, and you move on.

And that conclusion can be completely wrong. Not because a test lied. Every single green mark can be true. The problem is that the tests could not have told you anything else.

If you have done this, you are in very good company. I fell into a version of this myself, in my own published research, as you will hear in a minute. It is not carelessness. It is how test reports are built. A report is a list of checks, and every check says pass or fail. Nothing in that list talks about the comparison you actually care about.

And the usual advice does not help. People say, write more tests. Raise your coverage. Add another check. But more green is not more information. A thousand green marks that could never turn red tell you exactly as much as ten.

So how does it really work? Let me explain with a school.

Imagine two teachers. Each one teaches a class in a different way. At the end of the year, you want to know which way is better. So you give an exam. And every single student passes. Both classes. One hundred percent.

Does that mean the two ways of teaching are equally good? Of course not. Maybe the exam was so easy that nobody could fail it. Then the exam simply cannot see the difference between the classes. It is not a bad exam in the sense of wrong answers. It is a blind exam.

Now make it worse. Imagine that every student wrote their own exam. Their own questions. As many as they liked, with a minimum and no maximum. And then they sat it, after they had already practiced on it. One student wrote ninety-seven questions. Another wrote one hundred twenty-seven. Everyone passed.

Now ask the same question. Which class did better? And the honest answer is not "they are equal". The honest answer is: this cannot be compared. There is no shared exam. There is nothing to line up side by side.

That is exactly what happened in my experiment.

I ran a coding agent on twelve programming tasks. I tried six ways of clearing its context: a fresh session after every task, after every two, three, four, six, and one session for all twelve. Six repeats each. Thirty-six runs in total.

At the end of every run, the full test suite ran. Across all thirty-six runs, four thousand and eighty-six tests. Zero failures. Not one red mark, in any condition.

But in the same files, the cost was different. The most expensive of the six policies, on average, cost thirty-three and a half percent more than the cheapest one. To be fair about that number: it is the gap between the highest and the lowest of six averages, not a property of any single policy. Still, the setups were clearly not the same. And the tests said nothing about it.

Then I looked closer, and it got more uncomfortable. The agent wrote its own tests. In every run, from scratch. The prompts asked for a minimum number of test cases and set no maximum. So one run had ninety-seven tests, and another had one hundred twenty-seven. That is a thirty percent spread. By policy, the totals ranged from six hundred forty-one to seven hundred and one.

In the original report I had written that the test suite was held constant. It was not. That sentence was mine, and I corrected it.

And I want to be honest about one more thing. The tempting story here is that the longer sessions quietly did less work, and the green marks hid it. On average, the longer sessions did write somewhat fewer tests. But I checked, and the data does not support reading that as less work. The number of model calls fell much faster than the number of tests, and the amount of generated output actually went up. So I withdrew that claim. What remains is simpler, and it is enough: the amount of checking varied by up to thirty percent, and the report could not show it.

So now the term. Everything I just described is about one property. Whether a set of tests can tell two setups apart. In the paper I call it discrimination. Not of people. Of setups. Does your test suite discriminate between the things you are comparing?

Here is what follows from that.

First. A single check can be perfectly fine, and the whole set can still be blind. Nothing inside one test can tell you that. It is a property of the set.

Second. When nothing turns red, that does not mean the setups are equivalent. It means this suite did not see a difference. That is a statement about the suite, not about the setups.

Third. When the agent writes its own tests, you cannot even say "nothing changed". There is no same test in both runs to compare. The only honest answer is: incomparable.

So what I proposed is small. Every set of results should carry one field and one list. The field is called discrimination, and it has three values. Some: at least one check went from pass to fail, or back, between the setups. None: the checks were shared, and not one of them changed. And incomparable: the checks were not the same, so the question has no answer. The list names exactly which checks changed, because those are the checks carrying your result.

The idea came out of a discussion in the W3C Agent Conformance group, where colleagues had already worked out the per-check part: states, causes, and a mark for whether a pass could ever have been a fail. Nothing there is adopted yet; it is an open discussion. What was missing was the level above: the comparison.

And the reason the third value matters fits in one sentence. A record that cannot say incomparable will say none. And a reader will hear equivalent.

So what do you do tomorrow? Four steps.

Step one. Next to every green verdict, write the count. Not "all tests pass". Write "one hundred twenty-seven out of one hundred twenty-seven". If the counts differ between runs, you will see it right away.

Step two. Freeze your tests before you compare. Write them once, give each one a stable name, and do not let the agent write or edit the tests that judge it.

Step three. After a comparison, ask the one question from the start of this episode: which test changed? If the answer is none, write down "the suite did not tell these apart". Do not write "they are the same". If the tests were not the same in both runs, write "incomparable".

Step four. Prove that your suite can fail. Break something on purpose and watch a test go red. If nothing goes red, your suite is decoration.

Let me put it all together in three lines.

Green without a count is not a result.

The agent does not write the exam it is graded on.

Nothing changed does not mean the same.

The full paper, with the data and the corrections, is on Qeios: https://doi.org/10.32388/0BV3Z8. I am Evgenii Arsentev. Thanks for listening, and see you in the next episode.