Databricks · Databricks Data Engineer Associate
Bottlenecks in the Spark UI
Databricks Data Engineer Associate · Lesson 18 of 22 · 2:15:35
YouTube loads only after you press play. This chapter opens at its timestamp in the full course. Watch on YouTube ↗
In this lesson
Prepare for the Databricks Certified Data Engineer Associate exam. In this lesson: reading the Spark UI (jobs, stages, and tasks), spill, skew, and a single task or the driver as the bottleneck. Includes three original practice questions.
Preview the topics
- The task that builds Hollowbrook’s gold tables runs on a job cluster, and it has slowed down. The data engineer opens the Spark UI of the run. Where should they start?
- In the longest stage of the gold build, the 75th percentile task duration is 40 seconds, and the Max is 28 minutes. The stage page shows no spill statistics. What is the most likely cause?
- A Hollowbrook notebook computes a running total of sales for each store, with a window function ordered by date, but no PARTITION BY clause. Its longest stage has exactly one task. What should the data engineer change?