AWS · MLA-C02 · BETA
Evaluating ML and generative AI systems
MLA-C02 · Lesson 11 of 23 · 1:21:40
YouTube loads only after you press play. This chapter opens at its timestamp in the full course. Watch on YouTube ↗
In this lesson
Lesson 11 of the free MLA-C02 beta series (AWS Certified Machine Learning Engineer – Associate, domain 2, model development). In this lesson: reproducible experiments, baselines, and shadow tests, metrics for ML models and for text, human evaluation and LLM as a judge, and evaluating RAG with retrieve-only and retrieve-and-generate jobs. With three quick checks and a recap, and AnyCompany Mobility, a car sharing service, as the running example.
Preview the topics
- A new demand model must be compared with the current one on live traffic, with real latency. Customers must only ever receive answers from the current model. What should the ML engineer use?
- A model flags damaged cars from photos taken at each return. Missing a damaged car costs far more than an extra manual check. Which metric should the team optimize?
- The assistant gives some wrong answers, and the team must first find out whether the knowledge base returns the wrong chunks. It does not need to judge the generated text yet. Which evaluation fits?