Docs groundednessStable

Answers 180 setup questions using only public docs; penalises unsupported claims. Runs against onboarding-questions on Onboarding Guide v22.

Pass rate

96.8%

▲ 0.4 pts

Avg. score

0.95

Failing cases

6 / 180

Last run

198.0s

Pass rate by run

Last 10 runs · latest 6h ago by Mira Kowalski

A case passes only when every scorer passes.

Groundedness
97.2%
Groundedness≥ 0.90
Each claim is supported by a retrieved docs passage.
Helpfulness
95.6%
LLM judge≥ 0.75
Judge rates whether the answer unblocks the user.
Links to docs
98.9%
Regex/docs\//
Answer includes at least one docs link.

Showing 4 of 180 cases from the latest run · 1 failing

Failing only

	Case	Input	Groundedness	Helpfulness	Links to docs	Latency
	tc_0201	How do I invite my team and set roles?	Pass0.98	Pass0.91	Pass	860ms
	tc_0214	Can I use SAML SSO on the Team plan?	Fail0.31	Fail0.22	Fail	740ms
Input Can I use SAML SSO on the Team plan? Expected No, SSO requires Business; link docs/sso and pricing ActualFail Yes, enable it under Settings → Security. Groundedness: Docs state SSO is Business-only. Helpfulness: Wrong answer leads the user to a missing setting. Links to docs: No docs link.
	tc_0229	Where do I find my API key?	Pass0.99	Pass0.95	Pass	690ms
	tc_0233	What happens to runs if I pause an agent?	Pass0.96	Pass0.89	Pass	910ms