A data engineering team uses Auto Loader to ingest JSON files from an S3 bucket into a Delta table. After several days of operation, the team notices that new files are not being picked up. The Auto Loader job ran successfully with no errors. What is the most likely cause?

Question

Accepted Answer

C. Auto Loader is configured to use directory listing mode and new files were added to a prefix not included in the configured path. Auto Loader can operate in two modes: directory listing and file notification. In directory listing mode, Auto Loader scans the configured path. If new files were added to a different prefix or subdirectory not covered by the configured source path, they would not be detected. A permissions error would surface as a job failure, not a silent miss. Auto Loader has no maximum file count limit that causes silent pausing. Storage space issues would produce explicit errors.

A data engineering team uses Auto Loader to ingest JSON files from an S3 bucket into a Delta table. After several days of operation, the team notices that new files are not being picked up. The Auto Loader job ran successfully with no errors. What is the most likely cause?

Related Questions

Discussion