Anthropic · CCAR-P
Metrics and evaluation datasets
CCAR-P · Lesson 11 of 18 · 1:25:27
YouTube loads only after you press play. This chapter opens at its timestamp in the full course. Watch on YouTube ↗
In this lesson
Prepare for the Claude Certified Architect – Professional exam, on the exam guide version 1.0, effective July 2026. In this lesson: specific and measurable success criteria, accuracy, latency, cost, safety, and security metrics, building a realistic evaluation dataset with edge cases and an adversarial set, code-based, model-based, and human graders, grading agent outcomes and transcripts, capability and regression suites, and pass@k versus pass^k. Includes three original practice questions.
Preview the topics
- Harborline’s assistant reads customer emails, and the security team wants a measurable security metric. Which metric fits?
- Harborline’s extraction must return the policy number and the loss date exactly, and its letters must sound empathetic. Which grading approach fits?
- Harborline’s agent succeeds 80% of the time on a task, and customers rely on it every day, so each try must work. Which metric reflects this best?