Google releases Agile Deliberation for subjective image classifiers

Google Research released Agile Deliberation on Nov. 4, publishing a two-stage, LLM-guided workflow and code for building image classifiers around categories that depend on human judgment.

Agile Deliberation turns an evolving user definition into a classifier, then tests that definition against ambiguous images. Google evaluated the prototype in 18 live sessions lasting about 90 minutes each. The study, however, covered only nine non-expert participants and two concepts, limiting what its results establish.

The first stage, concept scoping, breaks a broad category into smaller “unit concepts.” The system proposes examples and positive or negative subconcepts, which users can include, exclude or discard to create an initial structured definition.

In concept iteration, the system looks for images near the definition’s gray areas. The research paper says the prototype generated borderline queries, retrieved roughly 50 to 100 images for each query, removed duplicates, clustered the results and selected clusters using an upper-confidence-bound rule. Users reviewed batches of at least five images, each focused on one ambiguity.

Users could label the images and explain disagreements with the classifier. An LLM then proposed revisions to the definition, and the prototype selected the candidate with the highest F1 score on labels accumulated across rounds. The implementation used Gemini 2.5 Pro for concept decomposition and Gemini 2.5 Flash for image classification, borderline-query generation and ambiguity mining, without additional training or fine-tuning.

The researchers recruited participants through an open call inside their organization. None had professional content-moderation experience. Each person used Agile Deliberation for one concept and a manual workflow for another, with the system order randomized. The two concepts were “Paid to Play” and “Healthy Food.”

In each session, participants spent about 45 minutes developing a definition, labeled 200 held-out images, completed a survey and took part in a 15-minute interview. Because each person used the two workflows on different concepts, the authors could compare subjective experience within participants but classifier performance only between participant groups.

In the paper’s main results, the researchers reported that Agile Deliberation improved average F1 by 10.5 percentage points over zero-shot classifiers and by seven points over Modeling Collaborator. Those gains came with a precision-recall tradeoff: Table 1 shows higher precision but lower recall than the zero-shot baselines for both concepts. The abstract separately reports a 7.5-point gain over automated-decomposition baselines.

All nine participants preferred Agile Deliberation, according to the paper. Of six survey measures, only two were marked statistically significant at p<0.05: lower effort to achieve good performance and fewer feelings of insecurity, stress or annoyance. The other measures, including frustration, mental demand and perceived success, were not marked significant.

The authors caution that the nine-person study may have limited statistical power and generalizability. They did not run component-level ablations or evaluate the full workflow across other generative-model families. Google’s public implementation includes an end-to-end Jupyter notebook and instructions for building a nearest-neighbor image index; its README says the project is not an officially supported Google product.