IBM and Kaggle launch maintained ScarfBench Java migration leaderboard

IBM and Kaggle launched a maintained ScarfBench leaderboard on Oct. 27 to compare model-and-coding-agent-harness combinations on enterprise Java framework migrations. IBM said Kaggle consulted on the evaluation design, independently reproduced and ran the benchmark on its platform, sponsored the compute and will add models as they are released.

The first evaluation spans 10 configurations across 204 directed migrations involving Spring, Jakarta EE and Quarkus applications. In IBM’s disclosed run, Gemini 4 Argon paired with Antigravity led the primary smoke-test metric at 75.2%, ahead of Claude Opus 5.5 with Claude Code at 70.5% and Claude Opus 5 with Claude Code at 69.8%. The smoke score awards partial credit and gives larger test suites more weight; Claude Opus 5.5 with Claude Code instead recorded the highest all-tests-pass rate, at 72.1%, versus 71.1% for Gemini 4 Argon with Antigravity.

ScarfBench evaluates each model together with its agent harness, which controls how the model interacts with the code repository and development environment. Its application-specific evaluation requires a migrated program to compile, deploy in a containerized target runtime and pass behavioral tests through its observable interfaces. The research paper describes 34 application families implemented as 102 framework variants, producing the 204 directed migration tasks.

Runtime behavior, rather than compilation, accounted for most of the classified failures in IBM’s October run. Of 745 failing runs, 535, or about 72%, were runtime behavioral failures; 152 involved compilation or packaging, and 58 involved startup or dependency-injection crashes. IBM also removed four tasks on which every configuration received zero behavioral-test credit for a sensitivity check. Scores increased, but the top-three order did not change.

The Kaggle collaboration is separate from IBM’s March release of the underlying ScarfBench benchmark, which already included reproducible tooling, validated workloads and a public comparison surface. The October launch adds Kaggle’s independent reproduction and ongoing maintenance of the leaderboard. Public model submission was not available at launch and remained on the roadmap. The ScarfBench paper cautions that its behavioral tests have no formal coverage guarantee, so passing every included test does not establish that a migrated application is defect-free.