spark技术分享PullupCorrelatedPredicates Logical Optimization
PullupCorrelatedPredicates is a base logical optimization that transforms logical plans with the following operators:
PullupCorrelatedPredicates is part of the Pullup Correlated Expressions once-executed batch in the standard batches of the Catalyst Optimizer.
PullupCorrelatedPredicates is simply a Catalyst rule for transforming logical plans, i.e. Rule[LogicalPlan].
|
1 2 3 4 5 6 7 8 9 10 11 12 |
import org.apache.spark.sql.catalyst.optimizer.PullupCorrelatedPredicates // FIXME // Demo: Filter + Aggregate // Demo: Filter + UnaryNode val plan = ??? val optimizedPlan = PullupCorrelatedPredicates(plan) |
PullupCorrelatedPredicates uses PredicateHelper for…FIXME
pullOutCorrelatedPredicates Internal Method
|
1 2 3 4 5 6 7 |
pullOutCorrelatedPredicates( sub: LogicalPlan, outer: Seq[LogicalPlan]): (LogicalPlan, Seq[Expression]) |
pullOutCorrelatedPredicates…FIXME
|
Note
|
pullOutCorrelatedPredicates is used exclusively when PullupCorrelatedPredicates is requested to rewriteSubQueries.
|
rewriteSubQueries Internal Method
|
1 2 3 4 5 |
rewriteSubQueries(plan: LogicalPlan, outerPlans: Seq[LogicalPlan]): LogicalPlan |
rewriteSubQueries…FIXME
|
Note
|
rewriteSubQueries is used exclusively when PullupCorrelatedPredicates is executed (i.e. applied to a logical plan).
|
Executing Rule — apply Method
|
1 2 3 4 5 |
apply(plan: LogicalPlan): LogicalPlan |
|
Note
|
apply is part of the Rule Contract to execute (apply) a rule on a TreeNode (e.g. LogicalPlan).
|
apply transforms the input logical plan as follows:
-
For Filter operators with an Aggregate child operator,
applyrewriteSubQueries with theFilterand theAggregateand its child as the outer plans -
For UnaryNode operators,
applyrewriteSubQueries with the operator and its children as the outer plans
