A data engineer wants to ingest a large number of existing Parquet files from an S3 bucket into a Delta table in a one-time batch operation. After ingestion, no new files will be added to that path. Which method is most appropriate?

Question

Accepted Answer

B. COPY INTO. COPY INTO is a SQL command designed for idempotent, incremental batch loading of files from cloud storage into Delta tables. It is well-suited for a one-time bulk ingestion scenario and is idempotent — running it multiple times will not re-ingest already-loaded files. Auto Loader with `trigger(once=True)` is also valid but adds streaming infrastructure overhead for a single-batch use case. Kafka and Change Data Feed are streaming/incremental change mechanisms not applicable to a static Parquet file directory.

A data engineer wants to ingest a large number of existing Parquet files from an S3 bucket into a Delta table in a one-time batch operation. After ingestion, no new files will be added to that path. Which method is most appropriate?

Related Questions

Discussion