AWS · AIP-C01
Evaluating FM outputs
AIP-C01 · Lesson 21 of 23 · 2:48:12
YouTube loads only after you press play. This chapter opens at its timestamp in the full course. Watch on YouTube ↗
In this lesson
Prepare for the AIP-C01 exam (AWS Certified Generative AI Developer - Professional). In this lesson: quality metrics for generative AI, Bedrock evaluations with programmatic jobs, judge models, and human workers, A/B and canary tests, cost-performance analysis, user feedback and annotation workflows, and RAG evaluation with retrieve-only and retrieve-and-generate jobs. Includes three original practice questions.
Preview the topics
- AnyCompany wants to compare three models on its own five hundred support questions, scored automatically for correctness and helpfulness, with an explanation for each score. What should it run?
- AnyCompany wants a new model to serve five percent of real support traffic first, and to serve everyone only if its quality and latency hold. What is this?
- AnyCompany’s RAG answers are often incomplete, and the team suspects the retrieval, not the model. Which evaluation isolates retrieval quality?