Source: arxiv.org/abs/2610.23284
Researchers have posted Annotrace, a workbench for labelling agent traces at the level of individual steps, on arXiv. The authors note that studies of why LLM agents fail usually begin with hand labelling, and each one builds its own throwaway tool. Observability platforms record traces without structured labelling, while text annotation tools have no concept of a step, delegation or tool call.
Annotrace reads six trace formats and converts each run into an ordered, nested list of steps. Annotators write their own labelling codebook. A frontier model proposes a starting set of labels from a sample of the user's traces, answers first, and is asked the same questions as the human, and the tool measures agreement between any two annotators, human or AI.
On a multi-agent question-answering corpus, Annotrace reproduces the 235 annotation units from an earlier manual effort exactly. Two models labelling the same traces agreed 91% of the time, at a kappa of 0.82. The paper is a four-page demonstration submission under review at the AAAI 2027 Demonstration Program.
Why it matters if you build agents
Developers who want to label failure modes in their agents can start from a step-level model instead of building a one-off tool. Support for OpenTelemetry GenAI, LangSmith and Langfuse traces means existing observability data may be reusable for annotation. The agreement figures come from a single corpus in a preprint that is still under review, so test them on your own agents before relying on them.
Get stories like this every weekday
One short email with the day's agent tooling news.