Lazy evaluation and the DAG

filter and select return DataFrames, so they are transformations. They only record steps in the logical plan. Spark reads no data until an action (count, show, write, collect) forces execution.

Source: https://spark.apache.org/docs/4.2.0/rdd-programming-guide.html#transformations

count, write, and collect return a value or rows, or they do I/O. So they are actions that launch jobs. withColumn returns a DataFrame, so it is a lazy transformation.

Source: https://spark.apache.org/docs/4.2.0/rdd-programming-guide.html#actions

groupBy/agg must bring together all rows that share a key. These rows generally live in different partitions, so the operation shuffles (wide). filter, withColumn, and select each compute an output partition from a single input partition (narrow).

Source: https://spark.apache.org/docs/4.2.0/rdd-programming-guide.html#shuffle-operations

Two wide transformations (the groupBy aggregation and the orderBy) create two shuffle boundaries. Stages = shuffles + 1 = 3. The read and the filter pipeline into stage 1.

Source: https://spark.apache.org/docs/4.2.0/sql-performance-tuning.html

Each Exchange node is a shuffle, which is a stage boundary. For a linear plan, the stage count is (number of Exchanges) + 1. Project and Filter are narrow. They pipeline within a stage.

Source: https://spark.apache.org/docs/4.2.0/web-ui.html

Transformations are lazy and Spark does not cache them by default. So each action recomputes the entire lineage. Call cache()/persist() (then an action) if you will reuse the DataFrame.

Source: https://spark.apache.org/docs/4.2.0/rdd-programming-guide.html#rdd-persistence

An Exchange is a shuffle. Rows move across partitions and machines. This is exactly like Redshift redistributing rows across slices when the join/group key does not match the DISTKEY.

Source: https://spark.apache.org/docs/4.2.0/rdd-programming-guide.html#shuffle-operations