A data engineer configures Auto Loader to ingest JSON files from an Azure Data Lake Storage Gen2 path. The engineer wants Auto Loader to automatically set up file notification services to reduce cloud costs and avoid directory listing overhead. Which Auto Loader file detection mode should they configure?

Question

Accepted Answer

B. File notification mode. Auto Loader supports two file detection modes: directory listing mode, which lists the contents of the input path to detect new files (default), and file notification mode, which automatically sets up file notification services on the cloud storage to push file arrival events without directory listing. Databricks documentation states that file notification mode makes file discovery much cheaper by avoiding directory listing altogether. Schema inference and trigger are not file detection modes.

Related Questions

Discussion