AI Code Writing Tools Locate Files But Not The Right Lines
SWE-Explore benchmark shows that AI code agents correctly identify files to modify but only find 14–19% of relevant code locations.
AI code agents are able to correctly identify files to modify but only find 14–19% of relevant code locations, according to the new SWE-Explore benchmark by an international group of researchers led by Shanghai Jiao Tong University. This is the first effort to assess code search independently of patch correctness over 848 problems in 203 open source projects and 10 programming languages.
What’s intriguing is that while AI code agents consistently rank the right file high, they struggle to find the right lines of code. Keyword matching performs “slightly better than random” for this task, and having a better model doesn’t seem to help. The problem is not one of intelligence but of exploration.
Why the missing lines break the fix
Researchers discovered that an agent will only successfully repair a bug if it has access to at least 50–75% of relevant code regions. If it doesn’t, even a good model will generate a very certain (but incorrect) patch because it doesn’t have all the information necessary to solve the problem. Researchers also found that lacking relevant information was worse than having too much irrelevant information, contrary to the intuition that we should reduce the amount of information given to our agents. In order to establish ground truth, researchers used verified patches generated by GPT-5.4, Gemini 3 Pro, Claude Sonnet 4.6 and Kimi K2.6.
This is the reason why you feel frustrated when using AI code agents to repair bugs: they sound certain, they edit the right file, but the bug persists. They weren’t being lazy, they just didn’t have access to the relevant lines of code. So if you’re using AI code agents, this means that you have to optimize for a different bottleneck now that models are getting smarter: you want to give your agent enough information before it starts writing so that it can identify the right context.
In my experience, this is also what happens when AI code agents fail: they try to solve a problem based on a limited understanding of the code base. But this is good news because there is something you can do about it: provide the agent with the information it wouldn’t normally have access to.
When an AI fix doesn't take, stop re-prompting the same way and instead hand it the context it's missing: paste the related files, the failing test, and the function that calls the broken one. Ask it to explain where the bug lives before it writes a patch — if the explanation is vague, it hasn't found the right lines yet, and any fix will be a guess. Treat the agent as a fast junior who's brilliant once it's looking at the right place, and blind until then.
Source: the-decoder.com
Free course
Stop reading about AI — start building with it
The free Claude Code course: your first site, tool or game — no coding. No upsells, no cross-sells — nothing to buy here.
Start free →▌ Related guides
Author
Evgenii Arsentev
PhD · Chief Executive Officer, digital health
Articles · Latest articles