A data engineer investigates a slow Spark stage in the Spark UI. Most tasks complete in 2–5 seconds, but one task takes over 10 minutes. Which observation in the Spark UI most strongly indicates that data skew is the root cause?

Question

Accepted Answer

B. A large variance in input data size across tasks, where one task reads significantly more data than the others. Data skew occurs when data is unevenly distributed across partitions. In the Spark UI task list for a stage, data skew is identified by a large variance in the Input Size / Records column: one task processes significantly more data than the median task. This directly explains why a single slow task outlier exists while most tasks complete quickly.

A data engineer investigates a slow Spark stage in the Spark UI. Most tasks complete in 2–5 seconds, but one task takes over 10 minutes. Which observation in the Spark UI most strongly indicates that data skew is the root cause?

Related Questions

Discussion