IBM's ALTK-Evolve tops τ²-bench retail with learned prompt guidelines on GPT-4.1

ALTK-Evolve uses the same model for guideline extraction and for the agent. It reviews the agent's attempts on training tasks and distills successes and failures into rules, such as how to authenticate a customer or confirm a transaction. The rules are added to the prompt for later conversations, while the benchmark's stock ReAct scaffold stays unchanged.

On held-out tasks with GPT-4.1, the guidelines raised both mean accuracy and Pass^4 in retail and airline. The authors found the best amount of guidance depends on the domain: retail gained most from all 292 retained guidelines, while airline's compact 14-guideline core gave the higher Pass^4 at 45%. Cost per retail conversation rose from $0.084 to $0.139, and the authors report slightly fewer assistant turns on matched runs.

On the full base split, the authors report that the airline agent using gpt-5.5 reaches 82.0% Pass^4, above frontier entries at 66–72%. The retail agent using GPT-4.1 reaches 67.5%, ahead of RAFT-30B-A3B at 61.4%, a model fine-tuned on synthetic retail data.

Developers can raise a stock model's agent performance on a benchmark by adding extracted rules to the prompt, without fine-tuning. The main cost is a larger system prompt on every call, so prompt caching and per-domain testing of guideline counts matter. Pass^4 can reveal reliability gains that average accuracy hides.