Task summary

The customer asks to close Green checking, keep Blue checking, and open a savings account. The agent closes Green and opens Bronze savings, but receives four failure-mode labels in the union reference. Full task in Sierra’s dataset: task_060, trial 1.

Reference

  • Confirmation or authorization failure. The assistant selected and opened a Bronze Account without obtaining the customer's confirmation of that specific savings account class, even though the opening procedure requires confirming the account selection.
  • Incorrect action parameter. The savings account was opened with the wrong account_class. Without eliciting the customer's requirements (withdrawal frequency, relationship bonus, minimum balance, daily compounding, ATM rebates, goals tracking), the agent passed 'Bronze Account' where the appropriate product was 'Silver Plus Account', producing an incorrect operational result that persisted to conversation end.
  • Protected information disclosure. The assistant revealed the internally retrieved customer name after receiving only one verification factor and before completing the required two-factor verification.
  • Retention or closure workflow bypass. The assistant closed the mid-tier Green checking account immediately despite the applicable three-day closure notice period.

Model outputs

Each cell is the model’s saved score for that failure mode. A score ≥ 0.5 predicts the label. “Extra” marks a false positive. “Missed” marks a false negative.

Failure mode Reference Jev Qwen 3.8 27B (no reasoning) GLM 5.3 Flash (low) Luna (none) Haiku (low)
Protected information disclosure Present 0.37 · Missed 0.00 · Missed 0.10 · Missed 0.98 0.25 · Missed
Confirmation or authorization failure Present 0.45 · Missed 0.80 0.70 0.85 0.50
Eligibility or prerequisite failure Absent 0.50 · Extra 0.90 · Extra 0.40 0.90 · Extra 0.85 · Extra
Retention or closure workflow bypass Present 0.61 0.85 0.90 0.98 0.70
Dependency ordering failure Absent 0.22 0.00 0.10 0.95 · Extra 0.10
Tool protocol misuse Absent 0.35 0.10 0.05 0.80 · Extra 0.05
Incorrect action parameter Present 0.23 · Missed 0.00 · Missed 0.05 · Missed 0.90 0.00 · Missed
Recommendation or optimization error Absent 0.20 0.00 0.15 0.75 · Extra 0.05
Unsupported or inaccurate information Absent 0.42 0.20 0.25 0.90 · Extra 0.05
TP / FP / FN 4 / 0 / 0 1 / 1 / 3 2 / 1 / 2 2 / 0 / 2 4 / 5 / 0 2 / 1 / 2
F1 1.000 0.333 0.571 0.667 0.615 0.571