Apple study finds tool-discovery gains shift with task complexity

Apple researchers found that language-model agents performed better when they retrieved tools before planning for single-tool requests. For within-category multi-tool requests, planning before retrieval worked better. The study compared the two discovery architectures across a registry of 46,984 API endpoints.

That crossover presents agent builders with an operational tradeoff. Planning first can split a complex request into separate functional needs, but the extra discovery call added 75% to 104% to inference-token use in the authors’ experiments and placed another sequential model round trip on the critical path. The comparison complements Apple’s recent research on how agent design affects benchmark outcomes by testing a separate choice: whether an agent should plan before or after tool retrieval.

Retrieve-then-Plan embeds the user’s raw request, searches the tool registry and passes the closest matches to a planner. Plan-then-Retrieve first asks a model to generate hypothetical tool descriptions, called abstract tool stubs. It then uses those stubs to search for real tools before sending the matches to the planner.

The paper’s main experiment used 500 ToolBench queries: 200 single-tool requests, 200 multi-tool requests within one category and 100 multi-tool requests spanning categories. Both approaches used GTE-base embeddings, an exact-search FAISS index, Gemini 2.5 Flash and a retrieval budget of five tools.

On single-tool queries, Retrieve-then-Plan recorded recall of 0.372, compared with 0.289 for Plan-then-Retrieve. Its pass rate was also higher, at 0.884 versus 0.840. The order reversed on within-category multi-tool queries: Plan-then-Retrieve reached recall of 0.188 versus 0.143 and a pass rate of 0.855 versus 0.747. The authors reported statistically significant differences for those measures.

The results were less conclusive for the 100 cross-category queries. Plan-then-Retrieve’s pass-rate lead, 0.820 versus 0.732, was statistically significant, but its recall of 0.172 versus 0.160 was not. In a four-run replication with gpt-oss-120b, the single-tool and within-category crossover held for precision and recall, while cross-category recall shifted weakly toward Retrieve-then-Plan.

The authors traced the split to the hypothetical stubs. On straightforward requests, generating stubs can remove product names and other terms that help a direct search find the right API. On complex requests, the model generated about 2.4 to 2.5 stubs per query, allowing separate searches for different parts of the task.

The pattern held across retrieval budgets from one to 20 tools and registry sizes from about 1,000 to 46,984 APIs. Recall declined for both approaches as registries grew, but Retrieve-then-Plan retained its advantage on single-tool queries and Plan-then-Retrieve retained its advantage on within-category multi-tool queries.

The added planning step raised estimated inference use from 226 to 396 tokens for single-tool queries, from 238 to 429 for within-category multi-tool queries and from 221 to 450 for cross-category queries. The authors estimated that sending single-tool requests through retrieval first and multi-tool requests through planning first would reduce total inference tokens by about 16% compared with always using Plan-then-Retrieve, assuming the appropriate query class were already known.

The evaluation compared retrieval and intended plans with ToolBench reference trajectories. It did not execute the APIs or test parameter accuracy and final-answer quality. The authors proposed the direct retriever’s top similarity score as a routing signal that would require no extra model call, but left empirical validation of that classifier to future work.