Study finds coding agent harnesses change cost more than benchmark scores

Liu and Han held the model constant and ran five models through Claude Code, OpenCode and mini-SWE-agent in their shipped versions. They tested on the 45 hardest tasks from SWE-bench Verified and a pool of 447 more, running some setups twice to measure noise. Runs were kept offline after early tests showed Deepseek and Qwen models pulling correct fixes from GitHub.

On the larger pool, Claude Code and mini-SWE-agent sit less than two points apart for both tested models, and a rerun of the same setup shifts results about as much as swapping harnesses does. OpenCode is the exception, trailing the other two by up to nine points, about half of which the authors tie to an output limit. Earlier reported gaps of more than 20 points almost always came from single runs.

Cost is where harnesses diverge. System prompts and tool descriptions are resent at every step, at roughly 16,600 tokens for Claude Code, about 7,000 for OpenCode and under 1,000 for mini-SWE-agent. For one model, a single task cost up to three times more depending on the harness. Thinking was the one case where harness choice changed outcomes: without it, mini-SWE-agent with Qwen3.6 dropped eleven points.

If you build coding agents, a heavier harness is unlikely to raise pass rates on its own, but it can multiply token spend, so measure cost per task before choosing one. Run the same setup twice before trusting a gap of a few points between harnesses. Whether reasoning is enabled can change which harness wins, so test that setting together with your chosen model.