Billion-Token Scale Trace Analysis: Jev vs LLMs

Large RL training runs produce billions of tokens across rollouts and checkpoint evaluations. Agents often have non-obvious failure modes, which require large-scale trace analysis to understand (we call these "gap analyses"). LLMs are a useful tool for analyzing traces at scale but both cost and out of the box calibration can prove to be bottlenecks to wide scale adoption. We find that Jev is particularly well suited to gap analysis workloads and is pareto optimal across cost and recall at certain decision thresholds.

Gap analyses drive multiple parts of our training workflow:

1. Curating post-training data: We identify recurring agent failures in base models or checkpoints and use them to create datasets that target those weaknesses
2. Scalable training run monitoring: We inspect traces for reward hacking and other failure modes to understand where model or environment behavior is unexpected
3. Improving production agents: Online RL for continual learning starts through a gap analysis of production traces to identify the failure modes to address through training

Developing a robust gap analysis system involves many technical challenges. For example, unforeseen failure modes can appear in the middle of a training run, so our detection system needs to dynamically identify and incorporate them into new clusters.

At this high volume, low cost classifiers make comprehensive gap analyses possible. The classifiers also need to generalize across runs, with a low false positive rate to limit unnecessary review and sufficient recall to detect meaningful failures. For this we have typically used LLMs like Luna or GLM, but we have found better performance in certain regimes with system one models like Jev. An overview of the trace analysis pipeline is shown in Figure 1.

Fig 1: Trace annotation at scale for agent failure modes. A sample establishes the failure taxonomy, classifiers assign the full corpus to leaves.

Auto Clustering: LLMs for map, Jev for reduce

Trace analysis starts by building clusters (labels) for annotations. We then use a map-reduce style pipeline to turn a collection of traces into a taxonomy (map), then annotating trace from a run into the taxonomy (reduce):

1. Map: Select sample traces from the RL run and prepare compact representations for token-efficient clustering. We use a low-cost model like Luna to compress long traces
2. Cluster: Recursively group the sampled traces into a taxonomy of possible annotations, with names and definitions for each category. This uses a frontier model like Kimi K3 or Sol to synthesize clusters
3. Annotate: Freeze the taxonomy, classify the full corpus against its leaves, and compute counts and outcomes by category. For this, we use low-cost classifiers like Jev, GLM 5.3 Flash, or Luna
4. Evolve: As new traces are clustered, existing nodes can split into more detailed subcategories, or new nodes can be formed. Intelligent node regrouping is done via similar models to clustering

When the annotation classes are already known, we can skip taxonomy discovery and classify traces directly.

Jev is particularly well suited for the large scale annotation workload. Rather than generating classification outputs autoregressively, Jev produces them in parallel, reducing sequential computation and enabling low-cost, low-latency classification. Since Jev is trained with Reinforcement Learning for Calibrated Decisions (RLCD)⌝, its probability outputs are well calibrated, allowing for efficient tradeoffs between precision and recall based on the confidence threshold.

AC2's native data pipelines can automate these stages, with judges⌝ returning structured labels and annotations⌝ attaching them to source traces.

Annotations surface failure modes

An annotation is a typed field attached to a trace. A trace can carry many annotations, each answering a different question. Common annotations include:

Some labels can be added deterministically which are much easier to apply at scale, like with regex or an environment state check. Others require semantic analysis: a tool call may execute successfully while violating a prerequisite stated earlier in the trace. Known failure modes can be checked from the start, while clustering helps discover behaviors we did not anticipate. Those discoveries become new labels that classifiers can apply across future rollouts.

Example: Customer support agent

To illustrate the annotation pipeline and benchmark the classifiers, we analyze customer support traces for recurring agent failures. We use the banking domain of τ³-bench⌝, where frontier models still have substantial headroom⌝. The benchmark asks an agent to resolve a simulated customer's request by finding the relevant banking policies and executing a sequence of tool calls, with success checked against the resulting account state.

For easy reproduction, we conduct a gap analysis on 148 of the published GPT-5.2 traces from Sierra⌝ across 69 banking scenarios. The corpus includes account changes, disputes, card replacements, and other difficult workflows.

Data processing

The initial sample uses 50 traces from distinct scenarios. Sol produces reference diagnoses for this sample.

Long tool outputs can dominate a trace. Before clustering, we can compress repeated payloads and summarize the interaction into a smaller representation of the agent's actions and their consequences.

Fig 2: Original traces → preprocessing → clustering → classification for annotation. Preprocessing combines tool-output compression and behavior summarization. Clustering builds the taxonomy used to annotate the full corpus.

Generating a failure mode taxonomy

The taxonomy generator uses Sol to read each sampled trace with its policy and reference context, producing a structured diagnosis with supporting events. We group those diagnoses into partial hierarchies, recursively merge overlapping categories, and assign the sample back to the merged taxonomy.

Each leaf has a name, definition, and explicit exclusions. For example, prerequisite or gate bypass covers acting despite an existing blocker, while harmful action sequencing covers taking actions in an order that creates a blocker.

Figure 3 shows the sample's 14 labels: 12 agent-failure tags and two other outcomes, with an example for each.

Fig 3: Agent failure mode taxonomy with sample traces on a calibration set in the τ³-bench banking domain.

Benchmarking classifiers: Jev vs LLMs

Once the taxonomy is frozen, we classify traces into the categories. We benchmark Jev (v1.13.0), Qwen 3.8 27B, GLM 5.3 Flash, Luna, and Haiku as low-cost classifiers, and use the union of Sol and Claude Opus labels as the reference (to reduce model-family bias). Each classifier receives the observable trace, policy, and category definitions. Reference diagnoses, proposed labels, and the benchmark's hidden reference actions are excluded from those inputs.

Each model outputs a floating-point score from 0 to 1 for each of 14 failure modes. A score at or above the decision threshold marks that failure mode as present. Figure 4 starts with micro F1 at a threshold of 0.5. Changing the threshold updates the points and Pareto frontier. Cost is shown per 1,000 individual annotations.

Fig 4: Annotation cost versus performance.

At threshold 0.5, Jev has the lowest cost among the models shown, while Luna medium reaches the highest micro F1. Notably, at threshold 0.20, Jev's recall rises to 85%, and it is Pareto optimal for micro F1 versus cost at this threshold (Figure 4).

Training runs can produce billions of generated tokens across tens of thousands of traces. Scaling up the cost estimates, to add one annotation to ten thousand traces in this dataset, Jev costs about $11 while Luna is $54 and Haiku 4.5 is $479. Overall, Jev provides notable cost savings.

Fig 5: Predicted scores versus observed label frequencies.

Among the classifiers shown, Jev is the most well calibrated, with the lowest ECE and Brier scores (Figure 5). Scores near 0.8 should correspond to positive reference labels about 80% of the time. Jev has lower calibration error than Luna medium (ECE 0.051 versus 0.154). Given Jev's low cost and relatively well-calibrated scores, we've found Jev is a useful as a first-pass filter at a high-recall decision point.

One limitation of Jev (v.1.13.0) is its 32k context length limit per classification query. For many moderate to long horizon takes, agent traces can span hundreds of thousands of tokens. This number can further increase if the agent is trained with compaction. Currently, Jev requires context summarization or splitting traces into segments to fit within its context window.

As an illustrative example, Figure 6 shows a sample task with the ground truth labels and per-model scores from the classifiers.

End-to-end failure mode analysis in AC2

This analysis was all conducted in AC2⌝, the Applied Compute platform. All the features used here are available in the platform: data pipelines built around datasets⌝, judges for failure-mode tagging⌝, and annotations attached to the original traces⌝. Ari automates these workflows⌝ through the AC2 SDK and CLI.

Large-scale trace analysis is useful to diagnose model behavior that is otherwise difficult to surface. Combining taxonomy discovery, inexpensive classifiers, and native AC2 annotations lets researchers identify recurring behaviors, inspect the evidence, and track whether changes to the model, environment, or reward address them.