26 AI Agent Test Methods Compared — All But One Degrade Performance

Dan Lu, a developer and engineer, ran a big experiment on 26 different ways to guide AI agents when writing code. He tried TDD, formal verification tools (Verus, Lean 4), fuzzing, property-based testing (Proptest, QuickCheck) and more. His goal was to implement the Zstd compression algorithm in Rust with GPT-5.6 Sol and Codex. He repeated each experiment 80 times, at two different difficulty settings.

1TDD and Formal Methods: Backfiring

Using TDD doubled the number of tests written, but reduced code quality. 67 out of 160 times, the tests were failing before the agent even started coding! Verus and Lean 4 looked good, but turned out to be about math, not code. Zero non-trivial bugs were detected in any of the 160 Kani runs.

2Surface Mimicry Instead of Understanding

Agents don’t know why you want to use a particular method. They just copy it. They use regular tests and adapt them to the specific framework. They write simple tests with the same input. And they create palindromic tests when you need to check the algorithm. Using the Hegel tool made things 26–41% more expensive without improving correctness.

3What Works: Default Instructions and a Human in the Loop

Without any special techniques, the agent did an average job. It even beat some “smart” approaches. Only one setup improved results: my own method. I had a well-structured testing strategy created by a human who knew the domain. There is a systemic flaw here. The top AI labs haven’t created the necessary RL environments so agents can learn how to test correctly. But testing is crucial!