A data engineer needs to run an automated ETL notebook every night. To minimize cost, which compute configuration should they use?

Question

Accepted Answer

B. A job cluster configured on a new cluster for the job. For automated, non-interactive workloads like scheduled ETL jobs, Databricks recommends using job clusters. A job cluster is created automatically when the job runs and terminated when it completes, so compute is only consumed for the duration of the job. An all-purpose cluster with auto-termination still incurs idle time between job start and auto-termination. Pools reduce startup time but the cluster still needs to be explicitly terminated. SQL warehouses are optimized for SQL analytics, not general PySpark ETL jobs.

A data engineer needs to run an automated ETL notebook every night. To minimize cost, which compute configuration should they use?

Related Questions

Discussion