Databricks · Databricks Generative AI Engineer Associate
Monitoring, Cost Control
Databricks Generative AI Engineer Associate · Lesson 17 of 17 · 2:07:01
YouTube loads only after you press play. This chapter opens at its timestamp in the full course. Watch on YouTube ↗
In this lesson
Prepare for the Databricks Certified Generative AI Engineer Associate exam. In this lesson: what to watch (volume, latency, error rate, tokens, quality, retrieval), inference tables and the usage and billing system tables, production monitoring with scorers on sampled traces, and controlling cost with rate limits, budgets, scale to zero, the right serving mode, and the smallest model that meets the bar. Includes three original practice questions.
Preview the topics
- Copperline’s team wants to review the exact requests and responses of last week’s conversations with the assistant, to build new test cases. Where should it look?
- One internal team runs a script that floods the assistant endpoint, and customers start to get slow answers. The internal team should keep working, at a lower volume. What should the team set up?
- An internal tool for Copperline’s finance team is used a few times a week. Its endpoint runs all the time, and costs too much, while a slow first answer would be acceptable. What should the team change?