A data engineer writes the following PySpark code to read CSV files from a Unity Catalog volume:

```python
df = (spark.read
.format(\csv\)
.option(\header\, True)
.option(\inferSchema\, True)
.load(/Volumes/main/raw/landing/\))
```

What does `inferSchema=True` cause Spark to do?

Question

A data engineer writes the following PySpark code to read CSV files from a Unity Catalog volume:

```python
df = (spark.read
  .format(\csv\)
  .option(\header\, True)
  .option(\inferSchema\, True)
  .load(/Volumes/main/raw/landing/\))
```

What does `inferSchema=True` cause Spark to do?

Accepted Answer

B. Performs an extra scan of the data to automatically determine column data types. When `inferSchema=True` is set on a CSV DataFrameReader, Spark performs an additional pass over the data to sample the values in each column and infer their data types automatically. Without `inferSchema=True`, all columns default to `StringType`. This is documented in Databricks PySpark documentation. It does not validate against a predefined schema, read from a checkpoint, or raise errors for nulls.

A data engineer writes the following PySpark code to read CSV files from a Unity Catalog volume:\n\n
`\ndf = (spark.read\n .format(\csv\)\n .option(\header\, True)\n .option(\inferSchema\, True)\n .load(\/Volumes/main/raw/landing/\))\n`
\n\nWhat does `inferSchema=True` cause Spark to do?

Related Questions

Discussion

A data engineer writes the following PySpark code to read CSV files from a Unity Catalog volume:\n\n\ndf = (spark.read\n .format(\csv\)\n .option(\header\, True)\n .option(\inferSchema\, True)\n .load(\/Volumes/main/raw/landing/\))\n\n\nWhat does inferSchema=True cause Spark to do?

Related Questions

Discussion

A data engineer writes the following PySpark code to read CSV files from a Unity Catalog volume:\n\n
`\ndf = (spark.read\n .format(\csv\)\n .option(\header\, True)\n .option(\inferSchema\, True)\n .load(\/Volumes/main/raw/landing/\))\n`
\n\nWhat does `inferSchema=True` cause Spark to do?