spark技术分享RewriteCorrelatedScalarSubquery Logical Optimization
RewriteCorrelatedScalarSubquery is a base logical optimization that transforms logical plans with the following operators:
-
FIXME
RewriteCorrelatedScalarSubquery is part of the Operator Optimization before Inferring Filters fixed-point batch in the standard batches of the Catalyst Optimizer.
RewriteCorrelatedScalarSubquery is simply a Catalyst rule for transforming logical plans, i.e. Rule[LogicalPlan].
|
1 2 3 4 5 6 7 8 9 10 11 12 |
import org.apache.spark.sql.catalyst.optimizer.RewriteCorrelatedScalarSubquery // FIXME // Demo: Filter + Aggregate // Demo: Filter + UnaryNode val plan = ??? val optimizedPlan = RewriteCorrelatedScalarSubquery(plan) |
evalExpr Internal Method
|
1 2 3 4 5 |
evalExpr(expr: Expression, bindings: Map[ExprId, Option[Any]]) : Option[Any] |
evalExpr…FIXME
|
Note
|
evalExpr is used exclusively when RewriteCorrelatedScalarSubquery is…FIXME
|
evalAggOnZeroTups Internal Method
|
1 2 3 4 5 |
evalAggOnZeroTups(expr: Expression) : Option[Any] |
evalAggOnZeroTups…FIXME
|
Note
|
evalAggOnZeroTups is used exclusively when RewriteCorrelatedScalarSubquery is…FIXME
|
evalSubqueryOnZeroTups Internal Method
|
1 2 3 4 5 |
evalSubqueryOnZeroTups(plan: LogicalPlan) : Option[Any] |
evalSubqueryOnZeroTups…FIXME
|
Note
|
evalSubqueryOnZeroTups is used exclusively when RewriteCorrelatedScalarSubquery is requsted to constructLeftJoins.
|
constructLeftJoins Internal Method
|
1 2 3 4 5 6 7 |
constructLeftJoins( child: LogicalPlan, subqueries: ArrayBuffer[ScalarSubquery]): LogicalPlan |
constructLeftJoins…FIXME
Executing Rule — apply Method
|
1 2 3 4 5 |
apply(plan: LogicalPlan): LogicalPlan |
|
Note
|
apply is part of the Rule Contract to execute (apply) a rule on a TreeNode (e.g. LogicalPlan).
|
apply transforms the input logical plan as follows:
Extracting ScalarSubquery Expressions with Children — extractCorrelatedScalarSubqueries Internal Method
|
1 2 3 4 5 6 7 |
extractCorrelatedScalarSubqueries[E <: Expression]( expression: E, subqueries: ArrayBuffer[ScalarSubquery]): E |
extractCorrelatedScalarSubqueries finds all ScalarSubquery expressions with at least one child in the input expression and adds them to the input subqueries collection.
extractCorrelatedScalarSubqueries traverses the input expression down (the expression tree) and, every time a ScalarSubquery with at least one child is found, returns the head of the output attributes of the subquery plan.
In the end, extractCorrelatedScalarSubqueries returns the rewritten expression.
|
Note
|
extractCorrelatedScalarSubqueries uses scala.collection.mutable.ArrayBuffer and mutates an instance inside (i.e. adds ScalarSubquery expressions) that makes for two output values, i.e. the rewritten expression and the ScalarSubquery expressions.
|
|
Note
|
extractCorrelatedScalarSubqueries is used exclusively when RewriteCorrelatedScalarSubquery is executed (i.e. applied to a logical plan).
|
