ScholarEvolve turns research papers into tested AI-agent harness changes

Researchers at UC Santa Barbara and Microsoft introduced ScholarEvolve, a framework that turns mechanisms from research papers into tested changes to the software around an AI agent without updating the underlying language model. The work was posted as an arXiv preprint on September 30.

In the authors’ experiments, the evolved harness raised Qwen3.5-27B task-goal completion from 69.0% to 81.4% on AppWorld Normal and from 49.6% to 63.6% on AppWorld Challenge. On Tau2-Bench Telecom, it raised GPT-5.4-mini pass@1 from 72.7% to 81.9% and pass@4 from 49.2% to 58.3%. The paper reports averages across three independent runs; the results have not been independently replicated in the opened sources.

An agent harness is the executable system around a language model. ScholarEvolve divides that system into five modules: tool interfaces, context management, skills, memories and agentic workflows. It audits failed agent trajectories to identify capability gaps, retrieves and deduplicates relevant papers, groups their mechanisms by module, and uses research and coding agents to propose and implement changes. Model parameters remain fixed throughout the process.

Candidate changes first undergo health checks covering imports, interface compatibility, artifact construction and reloading, and the production of valid actions. The system then evaluates individual changes and shortlisted combinations on validation tasks. A candidate replaces the current champion only when the lower bound of a paired bootstrap interval for its gain is above zero. The authors say evolution, validation and test tasks are disjoint, while persistent skill and memory artifacts are built only from evolution trajectories and frozen before validation and testing.

The search did not treat every paper-derived idea as an improvement. In the main Qwen3.5-27B AppWorld experiment, 17 of 20 candidates improved standalone development-set task-goal completion, eight had a 90% interval entirely above zero, and three sharply reduced performance and were excluded. The five-module combination beat its strongest individual module by 3.5 task-goal-completion points and 12.3 scenario-goal-completion points on the 57-task development set, according to the paper. Standalone rankings did not always predict the best combination.

The paper also presents a controlled reconstruction of three literature-update windows through August 2026. With Qwen3.5-27B, archived trajectories and derived artifacts held fixed, the best selected harness improved in each generation and reached 81.5% task-goal completion and 66.7% scenario-goal completion on the 168-task AppWorld Normal test set. The experiment modeled periodic literature-driven updates inside a benchmark, not unrestricted or live production deployment.

Some failure modes remained. The authors counted constraint violations in 89 failed episodes after evolution, compared with 91 before it, and incomplete retrieval in 70, compared with 77. Invalid response formats fell from 492 to 206.

The evidence is limited to the authors’ preprint experiments across AppWorld and Tau2-Bench Telecom. The paper says code is available through its linked ScholarEvolve repository, but the repository appeared empty when checked on October 22, preventing reproduction from the released implementation at that time.