ExtractPythonUDFs Physical Query Optimization
ExtractPythonUDFs
is a physical query optimization (aka physical query preparation rule or simply preparation rule) that QueryExecution
uses to optimize the physical plan of a structured query by extracting Python UDFs from a physical query plan (excluding FlatMapGroupsInPandasExec
operators that it simply skips over).
Technically, ExtractPythonUDFs
is just a Catalyst rule for transforming physical query plans, i.e. Rule[SparkPlan]
.
ExtractPythonUDFs
is part of preparations batch of physical query plan rules and is executed when QueryExecution
is requested for the optimized physical query plan (i.e. in executedPlan phase of a query execution).
Extracting Python UDFs from Physical Query Plan — extract
Internal Method
1 2 3 4 5 |
extract(plan: SparkPlan): SparkPlan |
extract
…FIXME
Note
|
extract is used exclusively when ExtractPythonUDFs is requested to optimize a physical query plan.
|
trySplitFilter
Internal Method
1 2 3 4 5 |
trySplitFilter(plan: SparkPlan): SparkPlan |
trySplitFilter
…FIXME
Note
|
trySplitFilter is used exclusively when ExtractPythonUDFs is requested to extract.
|