CCAR-P practice › Evaluation, Testing & Optimization · Question 63 of 110

Evaluation, Testing & Optimization

After an online test, a new retrieval configuration for an internal IT helpdesk assistant improves the primary metric, ticket deflection, by 6 points with statistical significance. The pre-declared guardrails were p95 latency under the SLA and cost per resolved ticket under budget. Both pass overall, but deflection for the French-language user segment fell by 9 points. What is the most appropriate action?

Show answer and explanation

Answer: D — Investigate the French regression and take the trade-off, with options, to stakeholders

(d) is right: a gain overall that hides a material regression in a segment is a trade-off for stakeholders to decide with the numbers, and the regression itself is a diagnostic lead (for example, French documents ranking poorly under the new retrieval settings). Shipping per segment or fixing the regression first are both reasonable options. (a) ignores that aggregates mask segment failures. (b) throws away a real improvement for the majority. (c) delays the decision without investigating the cause.