A data engineer streams incremental data from `system.billing.usage` into a Delta table for near-real-time cost monitoring. The streaming job intermittently fails with an error about unsupported source table changes. Which option should be added to the streaming read to prevent this failure?

```python
spark.readStream
.table("system.billing.usage")
.writeStream
.format("delta")
.option("checkpointLocation", "/chk/billing")
.start("/delta/billing_monitor")
```

Question

A data engineer streams incremental data from `system.billing.usage` into a Delta table for near-real-time cost monitoring. The streaming job intermittently fails with an error about unsupported source table changes. Which option should be added to the streaming read to prevent this failure?

```python
spark.readStream
    .table("system.billing.usage")
    .writeStream
    .format("delta")
    .option("checkpointLocation", "/chk/billing")
    .start("/delta/billing_monitor")
```

Accepted Answer

C. `.option(\skipChangeCommits\, 	rue\)`. System tables use Delta Sharing for data delivery, and Databricks may perform deletions on system table data (for example, when records expire after their retention period). Databricks documentation specifically recommends setting `skipChangeCommits` to `true` when streaming from system tables to avoid failures caused by delete commits in the source table. This option instructs the streaming engine to skip commits that contain only deletes.

Related Questions

Discussion