Microsoft · DP-750
Streaming ingestion
DP-750 · Lesson 11 of 19 · 1:24:02
YouTube loads only after you press play. This chapter opens at its timestamp in the full course. Watch on YouTube ↗
In this lesson
Prepare for the DP-750 exam (Microsoft Certified: Azure Databricks Data Engineer Associate). In this lesson: Spark Structured Streaming, checkpoints and triggers, Azure Event Hubs through its Kafka endpoint, Auto Loader with schema inference and evolution modes, and Lakeflow Spark Declarative Pipelines with streaming tables and append flows. Includes three original practice questions.
Preview the topics
- After a cluster restart, the Event Hubs stream processed all events from the beginning again, and wrote duplicates. What was missing from the stream?
- Outage files keep gaining new fields, and the Auto Loader stream must never stop, and never lose the unexpected data. Which schema evolution mode fits?
- The team wants to ingest outage files from a volume with the least code possible, and with checkpoints and schema evolution managed for it. What should it create?