Artificial Analysis and Google DeepMind launch benchmark for visual knowledge work
The Multimodal Knowledge Agent benchmark tests whether AI agents can turn dense visual files into professional deliverables, but private tasks and model-based grading limit comparisons beyond its setup.
Artificial Analysis and Google DeepMind launched the Multimodal Knowledge Agent benchmark to compare how well AI agents turn visually complex source files into finished professional work.
The scored evaluation covers 200 private tasks, split evenly across 10 sectors and spanning 83 occupations. The Google DeepMind technical report says each task includes between two and 45 input files, with an average of 7.3. Those files may include CAD blueprints, electronics schematics, radiographs, charts or geospatial maps.
Rather than simply answer a question, agents must combine information from the files into a standalone work product. The report names six deliverable extensions: .xlsx, .pptx, .pdf, .docx, .html and .png. It says 176 tasks, or 88% of the benchmark, require standalone files; the remaining 24 use text responses.
The benchmark uses 14,284 expert-written binary criteria, averaging 71.4 checks per task. Each criterion carries an integer weight from one to five. A task’s Weighted Satisfaction Rate, or WSR, is the share of total criterion weight that the grader marks as satisfied. The benchmark then averages task-level scores, giving each task equal weight. Google described a typical rubric as roughly 70 yes-or-no checks weighted 75% toward factual accuracy and 25% toward visual design.
The headline metric, Climb WSR, separates harder criteria from baseline “Regression” checks. The authors classify a criterion as Regression when all four models in a fixed prior-generation panel passed it in at least two of three runs; every other criterion is Climb. That split produced 6,396 Regression criteria across 180 tasks and 7,888 Climb criteria across all 200 tasks.
Artificial Analysis runs each model once per task through its open-source Stirrup agent harness. The setup provides a Linux sandbox, code execution, web search and page fetching, as well as image viewing for models that support image input. Each task is capped at 250 turns and eight hours.
A GPT-6 Luna model operating at medium reasoning grades each run in a separate, networkless sandbox. The judge receives the prompt, submitted files, final response, and each criterion’s description and weight, but not the original input files. Before returning a pass-or-fail verdict for each criterion, it can render pages, inspect internals such as spreadsheet formulas and run code against deliverables.
The report says the autorater agreed with human domain experts on 96.2% of criterion judgments in a 50-task study. The validation covered nine sectors, focused on criteria weighted three to five, and trimmed redundant passing checks to balance passes and failures. The authors estimate a 3.8% criterion-level error rate against human-majority verdicts, including 2.4% false passes and 1.4% false fails. They caution that the agreement result may not transfer exactly to low-weight formatting checks.
The 200 scored tasks—including their prompts, inputs and rubrics—remain private and are separate from the 10 downloadable showcase tasks. The public set demonstrates the format, but it cannot reproduce or expose the full leaderboard evaluation.
The grading also depends on the benchmark’s chosen models and machinery. The Climb-versus-Regression boundary comes from a fixed model panel and the same autorater’s verdicts. The report notes that false fails can label criteria as Climb even when the panel actually satisfies them. The scores therefore compare single runs within this task set, harness, tool configuration, judge and hardness split; they are not model-independent completion rates that can be directly exchanged with results from other benchmarks.
In the technical report’s four-model launch experiment, Claude Fable 5.2 scored 56.56% Climb WSR, Gemini 4 Carbon 54.34%, Claude Opus 5.5 53.01% and GPT-6 Astra 50.24%. Paired task-level bootstrap intervals left the top three statistically indistinguishable.
More news

Artificial Analysis tightens endpoint test as 24 of 103 trail references

Snowflake makes AI function evaluation and optimization generally available

MLCommons adds agentic and recommendation tests to MLPerf Training v6.1
