IBM Research says guidelines narrow AI-agent consistency gap

IBM Research introduced a method to measure and reduce inconsistent AI-agent behavior. Its authors report that targeted guidelines improved repeatable success on the AppWorld benchmark without lowering average accuracy in their same-task test. Across 168 tasks in the GPT-4.1 evaluation, Pass^5 rose from 53.0% to 69.0%, while Mean@5 increased from 77.4% to 81.0%.

The September 15 article addresses a reliability problem that average benchmark scores can hide. Mean@5 averages an agent’s results across five runs. Pass^5 counts only tasks the agent completes successfully in all five. The authors call the difference between those measures the consistency gap.

For the baseline GPT-4.1 ReAct agent, the gap was 24.4 percentage points. Based on the reported aggregate results, the same-task guidelines narrowed it to 12.0 points.

That focus on repeatability differs from other recent IBM Research work on model capabilities, including a theoretical comparison of quantum circuits and bounded language models. Here, the question is whether an agent can reproduce a result it has already achieved.

The accompanying technical report says the evaluation used AppWorld’s 168-task test_normal split, five runs per task and the benchmark’s standard grader. The researchers tested a ReAct agent with GPT-4.1 through Microsoft Azure and GPT-OSS-120B through AWS.

The Consistency Analyzer works as an offline, black-box diagnostic. It reissues each recorded prompt to the same model endpoint, samples alternative responses and measures variation among the parsed action choices. It does not need model logits, internal activations or access to the agent’s source code.

If a step on the decision path falls below a 0.85 consistency threshold, an LLM-based generator produces strategy, recovery or optimization guidelines. ALTK-Evolve stores them in episodic memory and retrieves them when the agent encounters the same task or a similar one.

The sampling budget requires some care. The introductory article describes one additional model call per decision step, requesting k completions with five as the default. For the reported experiments, however, the technical report says the default analysis budget was 30 samples for each parsed ReAct action pair at temperature 0.5.

IBM’s authors reported same-task Pass^5 gains of 22.9 percentage points on medium tasks, 14.3 points on hard tasks and 12.2 points on easy tasks. For GPT-OSS-120B, same-task Pass^5 rose 6.0 points from a 10.1% baseline. Guidelines transferred from similar tasks raised it by 8.7 points.

The GPT-4.1 transfer results came with a qualification. Guidelines generated from a sibling task variant increased aggregate Pass^5 by 13.0 points, but Mean@5 fell 0.8 points on the hard-task tier. The claim that average accuracy was preserved therefore applies to the same-task experiment, not every reported setting.

AppWorld’s benchmark paper describes a controlled environment with nine simulated everyday apps, 457 APIs and 750 tasks evaluated through changes in application state. IBM’s study used one split from that benchmark and two model endpoints. The results are author-reported: the opened evidence included no independent reproduction or audit, and performance on other benchmarks, production workloads, models and agent architectures remains unmeasured.

The authors linked their implementation through the public ALTK-Evolve repository, which lists both the technical report and the September 15 article.