A customer service AI is escalating 40% of all interactions to human agents because it escalates whenever Claude reports confidence below 60%. The system prompt says: "escalate if you are not confident." Why is this policy problematic and what should replace it?

Question

Accepted Answer

B. Self-reported LLM confidence is poorly calibrated and produces unreliable escalation decisions; replace with programmatic triggers: explicit human request, ambiguous policy identification, or no-progress detection. Self-reported confidence is an explicit CCA-F anti-pattern for escalation: LLMs may report low confidence for correct answers and high confidence for incorrect ones. Using it as the escalation signal creates both false positives (unnecessary escalations) and false negatives. Programmatic triggers — explicit human request, ambiguous policy, no progress after N attempts — are observable, measurable conditions that produce reliable escalation decisions. Tuning the threshold (Option A) attempts to fix an inherently unreliable mechanism.

A customer service AI is escalating 40% of all interactions to human agents because it escalates whenever Claude reports confidence below 60%. The system prompt says: "escalate if you are not confident." Why is this policy problematic and what should replace it?

Related Questions

Discussion