Databricks · Databricks Data Engineer Associate
Tuning Spark
Databricks Data Engineer Associate · Lesson 11 of 22 · 1:10:52
YouTube loads only after you press play. This chapter opens at its timestamp in the full course. Watch on YouTube ↗
In this lesson
Prepare for the Databricks Certified Data Engineer Associate exam. In this lesson: shuffle partitions and adaptive query execution, broadcast joins and collecting results to the driver, executor and driver memory, and measuring before you tune. Includes three original practice questions.
Preview the topics
- A daily aggregation of a few megabytes of sales runs on a small cluster, and the Spark UI shows two hundred tiny tasks for each shuffle, most of them taking milliseconds. Which setting should the data engineer look at first?
- A job on classic compute spills gigabytes to disk during a large join, and a teammate suggests setting spark.executor.memory in the notebook. What should the data engineer do instead?
- A data engineer changes shuffle partitions, the broadcast threshold, and the worker type at the same time. The job runs faster, but the cost goes up. What should they do next?