A data engineer is designing a Lakeflow Job with three tasks: Task A ingests data, Task B transforms it, and Task C validates the output. Tasks B and C must only run after Task A completes successfully. Task C must also wait for Task B to complete. How should the engineer configure the task dependencies?

Question

Accepted Answer

B. Set Task B to depend on Task A, and Task C to depend on Task B. Databricks Lakeflow Jobs supports task dependencies, where you can specify that a task should only run after its upstream tasks complete successfully. The correct dependency chain is: Task B depends on Task A (runs after A), Task C depends on Task B (runs after B). This creates a linear pipeline A → B → C. Running in parallel would ignore sequential data dependencies. Reversing dependencies is logically incorrect. Combining into one task loses individual task monitoring and retry capabilities.

Related Questions

Discussion