Each dot is one trace. Select a tag to see an example.
Fails to inspect, reconcile, or combine relevant records, documents, tool results, or risk signals before concluding, causing a missed discrepancy, incomplete audit, or false assessment of the current state.
The customer deposits a $1,500 check into a new savings account. The deposit tool reports success before the agent tells the customer whether it went through.
Check deposited successfully.
I’m **not seeing any posted or pending deposit yet**
Why this label: The agent denies seeing a deposit after the tool confirms success. The deposit is later confirmed in the balance; this is a recovered evidence-reading error.
Miscomputes a value, reconciliation, or correction, or ranks options incorrectly by misweighting known costs, benefits, thresholds, constraints, or tradeoffs.
The customer asks the agent to correct a monthly interest payment on a $144,000 balance. The account received $480 at a 4.0% annual rate; the agent identifies the applicable rate as 4.275%. Under the benchmark’s annual-rate-divided-by-12 calculation, the missing interest should be $33.
current_holdings: 144000.00
amount: 480.0
"expected_apy":4.275,"actual_apy":4.0,"amount_difference":30.0
Why this label: Using the benchmark’s monthly-rate calculation, $144,000 × (4.275% − 4.0%) / 12 = $33. The submitted correction is $30, leaving a $3 shortfall.
Applies a known policy-defined limit, count, entitlement, waiting rule, or eligibility criterion incorrectly to established case facts, producing the wrong classification or treatment.
Provisional credit is a temporary refund while a dispute is investigated. Starting point: 1 prior dispute, already credited. The customer now files 8 more disputes, and the agent marks 4 as eligible for provisional credit (events 45–48). These are counts of disputes, not dollar amounts.
The customer has not filed more than 2 disputes in the past 12 months
provisional_credit_given: True
Provisional Credit: ELIGIBLE - Credit will be applied within 2 business days.
You currently have **only 1 prior credit-card dispute in the last 12 months** on file with us, so that requirement is satisfied for provisional credit on the eligible cases.
Why this label: The reference reads the “not more than 2” rule as checking the running dispute count before each new filing: starting at 1, only 2 more disputes can qualify before the count reaches 3. The agent instead grants eligibility to 4 new disputes—2 too many under that reading. The rule does not explicitly explain batch handling, so this failure label depends on the reference interpretation.
Handles authentication or recovery incorrectly or unsafely, including demanding excess factors, disregarding valid factors, failing to log or act on completed verification, offering an inapplicable bypass, or disclosing stored identity data before verification.
Before filing the customer’s card disputes, the agent must verify identity using any two of four factors: date of birth, phone number, home address, or email. The customer has already supplied phone and email, which the agent acknowledges as confirmed; no third factor is needed.
2 of these 4
Since you’ve already confirmed the **phone number** and **email**, the remaining step is to confirm **one more** item
Why this label: The same reply acknowledges both supplied factors but demands a third. That contradiction is sufficient to illustrate the verification error.
Executes or recommends a consequential action without a mandatory eligibility, history, consent, retention, safety, or account-state check, or proceeds despite a pre-existing blocker or unsatisfied gate.
The customer asks to close the Crypto-Cash Back card. The retention procedure allows skipping retention if this card already has a closure-reason record from the past year. Otherwise, the agent must ask why the customer wants to close it and log the reason before proceeding. The history lookup returns no such record.
Ask the customer why they want to close their account. Then log it using log_credit_card_closure_reason_4521.
No closure reason records found for this credit card account.
Account cc_h1i2j3k4l5_crypto has been closed.
Why this label: The agent closes the card immediately after finding no reason record, without asking for or logging the reason in between.
Performs otherwise plausible workflow stages in the wrong order, violating a mandated sequence or causing an earlier action to block, delay, or undermine another requested action.
The customer wants to replace Light Blue checking with Purple checking and open Platinum Plus savings. Opening savings requires an active checking account held for at least 14 days. Light Blue satisfies that condition; a Purple account opened today does not. The agent therefore needs to open savings before closing Light Blue.
To open a **personal savings** account, you must already have at least one **active checking** account and you must have held your checking relationship for **at least 14 days**.
Bank account closed successfully!
Date Opened: 11/14/2025
Error: Account eligibility requirements not met.
Why this label: The agent knows savings requires an active checking relationship held for 14 days, but closes the seasoned account first. Its brand-new replacement cannot satisfy that requirement.
Fails to carry a known, requested, or already identified item through the complete workflow, including dropping one batch item, omitting a correction or report, forgetting a completed prior step, or unnecessarily deferring feasible work.
The customer asks the agent to dispute every rewards mismatch it identified, including an under-credit on a JetBlue purchase. Handling the other transactions does not complete this request: the JetBlue dispute must also be submitted and followed through.
Yes, please—let’s dispute all of the mismatches you listed.
One item still outstanding from the earlier mismatches: **JetBlue (txn_a8f1c2d3e403)** is still showing **3150 points**.
Why this label: After handling other transactions, the agent acknowledges JetBlue is still outstanding. The reference finds it remains unsubmitted and uncorrected at the end of the trace.
Selects the wrong operational branch, security treatment, escalation priority, transfer reason, or transfer path, including premature transfer or failure to follow a required transfer speedbump.
The customer wants to speak to a human. This benchmark has two different tools: initial_transfer_to_human_agent_0218 for the preliminary transfer procedure, and transfer_to_human_agents to complete the transfer. The reference policy requires calling the initial tool on each of the first three requests; the regular transfer tool is allowed only on request four. Here, the customer has asked twice, so the agent should still use the initial tool.
Can you transfer me please?
Please transfer me to a human agent.
transfer_to_human_agents
Why this label: On request two, the agent calls the tool that completes the transfer instead of the required preliminary tool. The error is skipping this benchmark’s prescribed sequence, not choosing to help the customer reach a human. The rule is supplied by the reference policy; the excerpts show the two requests and the actual tool used.
Performs a persistent action using an unsupported or incorrect target, amount, scope, reason, attestation, consent state, design, or status, including mutations based on unverified approvals or materially conflicting records.
The customer says their cash-back disputes have been approved and asks the agent to adjust rewards. The procedure requires checking the cash_back_disputes database for resolved disputes before changing rewards; the customer’s claim alone is insufficient. SUBMITTED means queued for review, not approved.
Status: SUBMITTED - Your dispute has been queued for review.
Look up the user’s resolved disputes in the cash_back_disputes database to find the transaction_id values that need rewards adjustments.
Transaction rewards updated successfully.
Why this label: The observed dispute status is only SUBMITTED. Despite the procedure requiring resolved disputes, the agent changes rewards without obtaining that confirmation.
Misuses the tool interface or discovery lifecycle by supplying unsupported parameters or noncanonical schema values, guessing an undiscovered tool, unlocking a tool before authorization, leaving an unnecessary unlocked tool unused, or repeatedly retrying an unavailable tool without recovery.
The customer needs to submit a cash-back dispute using submit_cash_back_dispute_0589. Its accepted arguments are user_id and transaction_id; it has no notes argument. The agent suggests optional text for dispute notes, and the simulated user adds that text as a notes argument when calling the tool.
you can paste this into the dispute notes (if prompted)
Error: Unexpected parameter: notes
Cash back dispute submitted successfully.
Why this label: The tool rejects notes and succeeds on retry without it. This is a recovered interface error; attribution to the agent is weaker because its suggestion was conditional (“if prompted”).
States an ungrounded policy, process, fee, exclusion, application method, troubleshooting step, timeline, navigation path, capability, automatic outcome, or assurance as though it were established by available documentation or tools.
The customer asks how long cash-back dispute review takes. The agent must answer from the bank’s documentation rather than invent a timeline. SLA means a stated service-time commitment; the agent acknowledges that the knowledge base provides none for this process.
the knowledge base doesn’t give a specific SLA for cash back disputes; in practice, reviews can take **anywhere from a few days to several weeks**
Why this label: The agent acknowledges that the knowledge base gives no SLA, then supplies a timeline anyway. The contradiction appears within this one excerpt.
Sends customer-facing output that is syntactically broken, truncated, or otherwise unusable, including malformed confirmations after an otherwise successful action or transfer.
The customer’s transfer to a human has succeeded. The next assistant message should confirm that result in readable text. In the message envelope below, the value of message is the customer-facing text; a lone opening brace is not a usable confirmation.
Transfer successful (reason: unconfirmed_external_communication).
{"message":"{"}
I’m only seeing a “{”
Why this label: The assistant sends only an opening brace, and the customer immediately reports the malformed output. The transfer itself succeeded.
The reference diagnosis attributes the recorded outcome to a simulator or grading issue rather than a supported agent failure.
The customer is choosing a savings account and credit card and needs to know whether the card offers paper statements. The agent cannot confirm that feature from the documentation. The system policy says to ask permission before transferring an unresolved question to a human; the agent does so and the customer agrees.
Because I can’t confirm paper statement availability for **credit cards** from the documentation I have, would you like me to **transfer you to a human agent**
Yes, please transfer me to a human agent to confirm **paper statements for the Silver Rewards Card** (and whether there’s any fee).
Why this label: The agent was penalized for not completing the transfer, but the recorded trace ends immediately after the customer consents, with no following assistant turn. The simulation appears to have stopped before the agent could act on that consent. The evidence therefore points to a simulator or grader issue; it does not establish that the agent refused or forgot to transfer.
The reference diagnosis found no supported agent failure in the observed interaction. This remains a provisional model judgment.
The customer requests a $2,500 increase on a premium card’s existing $5,000 limit. For that tier, the policy requires three consecutive on-time payment months and caps an increase at 50% of the current limit. Three months are recorded, and $2,500 is exactly 50% of $5,000. The reference also finds the other eligibility checks satisfied.
date_of_account_open: 03/20/2023 current_balance: $0.00 credit_limit: $5000.00
Consecutive on-time payments: 3
Previous Limit: $5000.00 - New Limit: $7500.00 - Increase: $2500.00
Why this label: The tool confirms an increase from $5,000 to $7,500, exactly as requested. These excerpts illustrate successful handling; the no-failure judgment relies on the full trace.