Microsoft · AI-500
Evaluation, human review, and continuous improvement
AI-500 · Lesson 11 of 16 · 1:31:26
YouTube loads only after you press play. This chapter opens at its timestamp in the full course. Watch on YouTube ↗
In this lesson
Prepare for the AI-500 exam (Microsoft Certified: Multi-Agent AI Solutions Expert). In this lesson: evaluating memory, knowledge, tools, and prompts, then the whole system, with Foundry's agent, process, and RAG evaluators, human evaluation templates in Foundry (preview), LLM-as-a-judge with calibrated, specialized judges, synthetic data generation, the agent optimizer (preview), and user feedback loops linked to traces, with regression baselines. Includes three original practice questions.
Preview the topics
- Contoso wants its compliance team to rate each refund agent response in Foundry with a thumbs up or down and a short comment. What should the team set up?
- Contoso’s judge model rates nine out of ten itineraries as excellent, but human reviewers disagree on many of them. What should the team do first?
- Travelers’ thumbs-down ratings on Contoso’s assistant sit in a spreadsheet that nobody links to the conversations. What should the design change?