AI Agent News/stories/together-ai-reports-agent-eval-gains-from-docs-skill-and-api-fixes.md

Infrastructure

Together AI reports agent eval gains from docs, skill and API fixes

Together AI ran coding agents against common developer tasks on its platform and used the failures to fix its docs, skill package and API error messages. The company says these fixes raised task completion from 75% to 97%, and it plans to open-source its eval tooling.

Source: www.together.ai/blog/how-we-do-agent-evals-at-together-ai

Together AI says more developers now use coding agents to build with its SDK and CLI, but it had no clear view of where those agents got stuck. It built a list of common tasks and ran each one under three setups: a baseline with no extra configuration, access to its docs through an MCP server, and its Together skill package. An LLM judge scored each run as pass, incomplete or fail, and the team also reviewed passing runs for wasted steps.

A GPU cluster pricing task exposed two problems. Sonnet 4.6 passed only 4 of 15 attempts, because a sample CLI output was read as live inventory and the skill linked to a marketing page instead of the pricing table. Two added sentences pointing the agent to the pricing table and the live regions lookup brought the task to 100% across the three models tested. A separate API error message also pointed agents to a route that does not exist, and the team has since corrected it.

The team also merged its 14 product skills into one unified skill. The first version passed 75% of runs, so the team shortened the router, replaced stale model names, and fixed bugs in sample code. The final version passed 70 of 72 runs, or 97.2%, at the original cost per run.

Why it matters if you build agents

If you build agents that call Together, a weaker model may fail where a stronger one works around the same gap, so test across capability levels rather than only with the most capable model. CLI help text, API error messages and skill files are often the first things an agent reads when it gets stuck, so keep them accurate. Together says it will release the eval package it uses next week, which lets you run your own task lists and compare configurations.

Get stories like this every weekday

One short email with the day's agent tooling news.

Subscribe