Anthropic · CCAR-P
A/B tests and diagnosis
CCAR-P · Lesson 12 of 18 · 1:31:18
YouTube loads only after you press play. This chapter opens at its timestamp in the full course. Watch on YouTube ↗
In this lesson
Prepare for the Claude Certified Architect – Professional exam, on the exam guide version 1.0, effective July 2026. In this lesson: iterating one variable at a time, A/B tests in production with primary and guardrail metrics, gradual rollouts and shadow mode, and diagnosing prompt failures, hallucinations, model mismatches, and stop reasons, then turning every fix into a regression test. Includes three original practice questions.
Preview the topics
- One Harborline release changes the prompt, the model, and the retrieval settings at once, and quality rises four percent. Which change caused the gain?
- A new version of Harborline’s claim summaries has passed the offline evaluations. How should the team test it with live traffic?
- A teammate upgrades the model ID to Opus 5.5, and every request that forces a specific tool now fails with error 400. What is the diagnosis?