Catalyst and the physical plan

Pushdown, pruning, filter reordering and constant folding are rule-based rewrites. Catalyst applies them to the analyzed plan to produce the optimized logical plan. The physical plan then carries the results (for example, PushedFilters on the scan).

Source: https://spark.apache.org/docs/4.0.0/sql-programming-guide.html

A native column predicate on a Parquet source normally appears in PushedFilters. Suppose the list is empty for a filter that you expected. Usually a Python UDF or a non-pushable function wraps the column. This makes the predicate opaque to Catalyst.

Source: https://spark.apache.org/docs/4.0.0/sql-data-sources-parquet.html

You see column pruning as a narrow ReadSchema on the scan node. Spark reads only the referenced columns. Parquet is columnar, so this is a real I/O reduction, not a cosmetic one.

Source: https://spark.apache.org/docs/4.0.0/sql-performance-tuning.html

'formatted' prints the physical operators as a compact numbered tree plus a details block (ReadSchema, PushedFilters, output). 'cost' adds row and size estimates. 'codegen' dumps the generated Java. 'simple' prints one line for each operator.

Source: https://spark.apache.org/docs/4.0.0/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.explain.html

Pushed predicates let the Parquet reader skip whole row groups. It uses the column min/max stats in the footer. So Spark never reads or decodes those bytes. This is the same idea as Redshift zone maps that skip blocks.

Source: https://spark.apache.org/docs/4.0.0/sql-data-sources-parquet.html

When the filter moves below the join, the join inputs become smaller. Catalyst applies this as a rule-based rewrite in the optimized logical plan. A warehouse planner does the same when it pushes a WHERE clause before a join.

Source: https://spark.apache.org/docs/4.0.0/sql-performance-tuning.html

explain only plans. It shows the parsed, analyzed, optimized, and physical plans without it running the query. Spark reads no data until an action triggers execution of the physical plan.

Source: https://spark.apache.org/docs/4.0.0/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.explain.html