Databricks · Databricks Generative AI Engineer Associate
Evaluating Your App
Databricks Generative AI Engineer Associate · Lesson 16 of 17 · 2:01:10
YouTube loads only after you press play. This chapter opens at its timestamp in the full course. Watch on YouTube ↗
In this lesson
Prepare for the Databricks Certified Generative AI Engineer Associate exam. In this lesson: running mlflow.genai.evaluate with an evaluation set, scorers, traces, and spans, the built-in judges that need no ground truth and those that do, custom scorers with the scorer decorator and make_judge, and expert feedback in the Review App to label traces and align your judges. Includes three original practice questions.
Preview the topics
- Copperline wants to know whether the chunks that come back hold everything needed to answer each question. Its evaluation set has expected facts for every question. Which judge fits?
- Every answer must include a page number from the manual, checked in development and in production monitoring. A simple pattern match can decide. What should the team write?
- Copperline built a custom judge for helpfulness with make_judge, but it often disagrees with the support leads, who know the products best. The leads have already labeled 200 traces. What should the team do?