Cohere tests the cost-performance trade-off in coding-agent teams

Cohere tested how the roles, models and size of coding-agent teams affected ProgramBench results. The company reported that a strong lead assigning cheaper workers only to planning approached the test-pass score of an all-frontier team at substantially lower inference cost. A Sol lead with 16 Luna planning workers reached a 61.8% average hidden-test pass rate at a total cost of $155, compared with 63.2% and $510 for an asynchronous all-Sol team that delegated planning, implementation and verification.

The 70% cost reduction applies only to those two configurations in Cohere’s controlled benchmark study. Each fixed-size setup with two, four, eight or 16 Luna workers was run once. By contrast, Cohere said the study’s main configurations were generally run three times across all 200 ProgramBench tasks. The evidence does not establish that the performance or cost ordering transfers to production software development, other agent harnesses, other models or future pricing.

For the study, Cohere built a minimal agent-orchestration harness in which agents shared a workspace. A lead received the task, delegated separate subtasks and combined the workers’ reports. Each worker initially saw only its assigned subtask. The experiments varied where workers were used, whether the lead waited for them or continued asynchronously, which models filled the lead and worker roles, team size and whether agents could message one another directly.

ProgramBench asks agents to reconstruct 200 programs from an executable and documentation, without access to source code. Hidden behavioral tests compare the rebuilt program with the reference executable. Cohere scored each task by the share of those tests passed, then averaged that share across tasks. The benchmark’s maintainers identify fully resolved programs as the primary metric and caution that average test-pass rate can overstate completion because every task includes simple tests. Cohere said it used the finer-grained measure to compare changes in team design.

Using the same minimal prompt and Sol model, Cohere reported a 61.8% reward for a blocking team and 52.1% for a solo agent. With a structured workflow covering planning, implementation and verification, the blocking team scored 63.1%, compared with 57.7% for the solo agent. According to the company, the four all-Sol team configurations cost four to seven times as much as the corresponding solo setups.

The clearest efficiency result came from controlling which stage received extra agents. Among asynchronous all-Sol teams, delegating only exploration and planning produced a 63.6% reward at $249. Using workers across all three stages produced 63.2% at $510. Measured as reward gained over the solo structured-workflow agent per $100 of added cost, planning delivered 3.4 points; implementation, the next-best stage, delivered 1.7 points.

Cohere attributed planning’s advantage to ProgramBench’s black-box design: multiple workers could probe different behaviors of the reference executable before the lead wrote the replacement. A Sol lead with cheaper Luna workers assigned only to planning scored 60.5%, against 63.6% for the all-Sol planning team. Cohere calculated that the mixed team retained 47.5% of the all-Sol team’s gain over the solo baseline while incurring 13.9% of its added cost.

Lead strength also mattered in that setup. With Luna planning workers held constant, Cohere reported rewards of 42.5%, 51.6% and 60.5% as the lead changed from Luna to Terra to Sol. The corresponding solo scores were 37.4%, 47.2% and 57.7%. The Terra-led team still trailed a solo Sol agent.

A secondary test reversed the ordering. On the 16-task Commit0-Lite benchmark, where repositories, documentation and tests specify what to build, implementation was the most effective stage to delegate in eight of 10 team setups. Cohere called the result suggestive rather than conclusive because capable solo agents already passed every test on 12 of the 16 tasks.