Coding-agent harnesses can triple costs without clear accuracy gains, study finds

Switching the software harness around a fixed coding model changed average per-task costs by as much as threefold, Shandong University researchers found. Most measured accuracy differences, however, did not clearly exceed the variation seen when identical setups were rerun.

Yangze Liu and Zhongyi Han submitted the 22-page preprint on Oct. 3. It compared Claude Code, mini-SWE-agent and OpenCode on SWE-bench Verified. The clearest result was economic, not evidence that harnesses never affect accuracy: system prompts, tool definitions and interaction patterns repeatedly add tokens to model calls, changing the bill even when the model and tasks remain fixed.

A harness is the software layer that turns a chat model into a coding agent. It provides the system prompt and tools, feeds tool output back to the model and manages the growing conversation. The researchers pinned production versions of all three harnesses, then ran five models through each one on 45 tasks drawn from the benchmark’s hardest annotated set. They also tested two locally served Qwen models on the remaining 447 offline-runnable tasks. Claude Opus 5 served as an additional anchor only in Claude Code, not in the cross-harness comparison.

The cost difference began with the fixed material sent on every call. The paper measured preambles of 16,581 tokens for Claude Code, 7,025 for OpenCode and 829 for mini-SWE-agent. Each step resends the conversation transcript, so that starting weight is incurred repeatedly. Context growth per step remained within a factor of 1.5 across harnesses, making preamble size and model-call count the main observed cost drivers.

For GLM-5.3-Flash, the authors calculated mean per-trial costs of 1.81 Chinese yuan with Claude Code, 0.97 yuan with mini-SWE-agent and 0.56 yuan with OpenCode. OpenCode carried less input per step and stopped after fewer calls. That ordering did not hold for every model: on DeepSeek-V4-Flash, mini-SWE-agent took more steps than Claude Code and moved the most tokens despite its lighter preamble.

Cache pricing changed the total bill, but not the harness ordering by token use. The paper reports that 96% to 99% of input tokens were cache hits in the two vendor-priced model arms. Repricing each model’s tokens with the other vendor’s cache rates preserved the ranking of the three harnesses.

The accuracy evidence was more limited. On the 447-task pool, Claude Code and mini-SWE-agent were statistically equivalent within a five-percentage-point margin for both Qwen models tested there. Across the 45 hard tasks, the three harnesses differed by two to five solved tasks per model, but every paired comparison interval included zero and the leader changed by model.

Reruns explain why those small differences were inconclusive. Harness swaps and identical-configuration reruns each changed outcomes on a median 13% of hard-set tasks. At the observed discordance rate, the paper’s power analysis estimated that 45 tasks would detect a 12.9-point pass-rate gap only half the time and could not detect any gap with 80% power. A 447-task comparison could detect a 5.2-point gap with 80% power.

One accuracy loss was statistically distinguishable. On the larger pool with Qwen3.8-27B, OpenCode trailed Claude Code and mini-SWE-agent by about eight to nine percentage points. Thirty-three OpenCode trials ended at an unrecovered output cap, compared with one for Claude Code. Removing the capped OpenCode trials roughly halved the gap. But a separate recovery experiment was not statistically distinguishable from no effect, so the study did not establish that recovery logic alone caused the full difference.

The results focus on the software around fixed models, a design space also examined in research on changing agent harnesses while keeping the model fixed. The authors limited their conclusions to one main benchmark and described a one-model Terminal-Bench 2.1 run only as a transfer check. The work is a preprint, and the supplied sources do not establish peer review or independent replication. The authors released a trial table, analysis scripts and Harbor job files; they said full mini-SWE-agent and OpenCode trajectories and Claude Code tool-call records were forthcoming.