AWS · MLA-C02 · BETA
Monitoring models and drift
MLA-C02 · Lesson 19 of 23 · 2:38:07
YouTube loads only after you press play. This chapter opens at its timestamp in the full course. Watch on YouTube ↗
In this lesson
Lesson 19 of the free MLA-C02 beta series (AWS Certified Machine Learning Engineer – Associate, domain 4, operating, monitoring, and securing). In this lesson: SageMaker Model Monitor and its four kinds of monitoring, A/B tests with production variants, anomalies in ML workflows, and monitoring foundation models. With three quick checks and a recap, and AnyCompany Mobility, a car sharing service, as the running example.
Preview the topics
- The fare model predicts the price of each trip, and the real price is known two days later. The team wants to know when the model's error grows, not just when the inputs change. Which monitoring should the ML engineer set up?
- The nightly feature pipeline usually processes about two million trips. Last night, it succeeded with only forty thousand rows, and nobody was warned. What should the ML engineer add?
- Users say the support assistant's answers got worse over the last month, but the latency and error metrics look normal. How should the ML engineer track answer quality over time?