Databricks · Databricks Data Engineer Associate
Auto Loader
Databricks Data Engineer Associate · Lesson 5 of 22 · 26:12
YouTube loads only after you press play. This chapter opens at its timestamp in the full course. Watch on YouTube ↗
In this lesson
Prepare for the Databricks Certified Data Engineer Associate exam. In this lesson: how Auto Loader and the cloudFiles source work, the directory listing and file notification modes, schema inference and schema hints, schema evolution modes, and the rescued data column. Includes three original practice questions.
Preview the topics
- The app events folder now holds millions of files, and each Auto Loader run spends more time listing the folder than loading data. What should the engineer change?
- The app team adds new fields every few weeks, and the bronze table must get them as new columns, with the job carrying on after a short restart. Auto Loader runs with no schema provided. Which evolution mode fits?
- A store's file sends a quantity as the text “twelve”, in a column of type integer. Auto Loader runs with its rescued data column. What happens to that row?