Ticket routing accuracyImproved

Checks category, priority and reply quality against 240 labelled tickets from Q3. Runs against support-tickets-q3 on Support Triage v15.

Pass rate

94.2%

▲ 2.1 pts

Avg. score

0.91

Failing cases

14 / 240

Last run

412.0s

Pass rate by run

Last 10 runs · latest 2h ago by Mira Kowalski

A case passes only when every scorer passes.

Category match
96.7%
Exact match= label
Predicted category equals the human label.
Reply quality
93.3%
LLM judge≥ 0.80
Atlas-4 judge scores tone, accuracy and next step on a 0–1 scale.
Latency budget
97.9%
Latency budget≤ 2.5s
End-to-end time stays inside the support SLA budget.

Showing 6 of 240 cases from the latest run · 3 failing

Failing only

	Case	Input	Category match	Reply quality	Latency budget	Latency
	tc_0012	I was charged twice for the Team plan in October. Can you refund one?	Pass	Pass0.94	Pass	1.8s
	tc_0031	SSO login loops back to the sign-in page for everyone in our org since this morning.	Fail	Fail0.41	Pass	2.2s
Input SSO login loops back to the sign-in page for everyone in our org since this morning. Expected category: outage · priority: urgent · link status page, escalate to on-call ActualFail category: authentication · priority: high · suggested clearing cookies Category match: Expected outage; org-wide symptom should trigger outage routing. Reply quality: Advice is generic and does not escalate an org-wide incident.
	tc_0047	How do I export all my workspace data before we cancel?	Pass	Pass0.88	Pass	1.6s
	tc_0058	Your API returns 500 on /v2/batches when the payload has more than 500 items.	Pass	Pass0.90	Fail	2.7s
	tc_0063	Can we get an invoice in EUR instead of USD from next month?	Pass	Pass0.92	Pass	1.5s
	tc_0077	lol your bot just told me to restart my router. this is a web app	Pass	Fail0.72	Pass	1.3s