CCAR-P practice › Evaluation, Testing & Optimization · Question 63 of 110
Evaluation, Testing & Optimization
After an online test, a new retrieval configuration for an internal IT helpdesk assistant improves the primary metric, ticket deflection, by 6 points with statistical significance. The pre-declared guardrails were p95 latency under the SLA and cost per resolved ticket under budget. Both pass overall, but deflection for the French-language user segment fell by 9 points. What is the most appropriate action?
Show answer and explanation
Answer: D — Investigate the French regression and take the trade-off, with options, to stakeholders
(d) is right: a gain overall that hides a material regression in a segment is a trade-off for stakeholders to decide with the numbers, and the regression itself is a diagnostic lead (for example, French documents ranking poorly under the new retrieval settings). Shipping per segment or fixing the regression first are both reasonable options. (a) ignores that aggregates mask segment failures. (b) throws away a real improvement for the majority. (c) delays the decision without investigating the cause.
Keep going
More Evaluation, Testing & Optimization questions
- A team is comparing two prompt variants for a claims-summary feature over a four-week…
- A pharmaceutical company's RAG assistant answers questions about drug-interaction…
- A procurement agent's responses intermittently end mid-sentence when it summarizes long…
- A logistics company routes all customer email through a large, capable model that…
- An internal policy assistant started giving inconsistent answers about expense limits a…