UDFs and the Pandas API

4 h
requires SPK-05

From your world. dbt Python models, and the pandas you reach for out of habit

Practice quiz with instant feedback. For a scored run that counts toward your local progress, use uv run de quiz SPK-09.

2Why is a scalar Python UDF usually slower than the equivalent built-in function? AIt runs on the driver instead of the executors. BIt serializes every row to a Python worker and back, and it is an optimizer fence (no pushdown/codegen through it). CIt forces the whole DataFrame to be cached. DIt always triggers a shuffle.

3You need custom Python logic over a numeric column, but you want to avoid BatchEvalPython. Which option transfers data as Arrow batches and shows as ArrowEvalPython? Afunctions.udf(fn, 'double') with the default (non-Arrow) path BA pandas UDF (Series -> Series), e.g. @pandas_udf('double') CCollecting to the driver and using a Python for-loop DA SQL string in spark.sql()

5What is the main advantage of the Pandas API on Spark (import pyspark.pandas as ps) over real pandas? AIt is 100% API-compatible with pandas. BIt runs distributed and lazily on Spark, so you keep pandas ergonomics and you do not pull all data onto one machine. CIt never shuffles. DIt replaces the need for the DataFrame API entirely.

6A teammate converts order amounts to USD. The teammate joins a daily fx_rates table and multiplies. Another teammate writes a Python UDF that looks up the rate for each row. Both return the same numbers. Which statement is TRUE? AThey are equivalent in performance. BThe join+multiply stays in Catalyst (no BatchEvalPython). The per-row UDF adds a Python round trip and blocks pushdown. CThe UDF is faster because it avoids a join. DOnly the UDF can be correct.

7Which of these are legitimate reasons to prefer a pandas/Arrow UDF over a plain Python UDF? (select all) (select all that apply) AData crosses the JVM/Python boundary once per Arrow batch instead of once per row. BYou can spread an expensive one-time setup across batches with the iterator form. CIt guarantees that the result is always correct regardless of logic. DIt operates on vectorized pandas Series rather than scalar Python values.