Spark SQL and data sources

5 h
requires SPK-03

From your world. dbt {{ ref() }} and marts vs. temp views and SQL on files

Read a dirty CSV with an explicit schema

The customers feed also arrives as CSV. It has a header row. NULLs appear as empty strings. If you read it with inferSchema, it costs an extra scan and risks wrong types. The production-correct way is to declare the schema, exactly like typing a dbt source.

Implement read_customers_csv(spark, csv_path) in solution.py. Return a DataFrame that reads the CSV at csv_path with these settings:

an explicit schema (no inferSchema) with these columns and types:
- company_id → string
- customer_id → long (bigint)
- first_name → string
- last_name → string
- email → string
- signup_date → string
header = true (skip the header row),
nullValue = "", so an empty email cell becomes a real NULL, not an empty string.

The test compares your result against the Parquet version of the same table (the typed source of truth). It uses an independent read. If your types or NULL handling are wrong, the comparison fails. The test ignores row order and column order.

Starter code

solution.py

"""SPK-04 / 03-read-csv-schema — starter. See README.md."""


def read_customers_csv(spark, csv_path):
    # TODO: build an explicit StructType and read the CSV with header + nullValue=''.
    raise NotImplementedError

Hints

Hint 1

Build a StructType with the right types: company_id/first_name/last_name/email/signup_date as StringType, customer_id as LongType. Then spark.read.schema(schema).

Hint 2

The CSV has a header row and renders NULLs as empty strings. Use .option('header', True) and .option('nullValue', '') so empty email becomes NULL, not ''.

Hint 3

Do NOT use inferSchema. Return the DataFrame as read; the test compares it to the Parquet version of the same data (the typed source of truth).

Run it locally

uv run de start SPK-04/03-read-csv-schema
# edit workspace/SPK-04/03-read-csv-schema/
uv run de test