spark技术分享ResolveSubquery Logical Resolution Rule
ResolveSubquery is a logical resolution that resolves subquery expressions (ScalarSubquery, Exists and In) when transforming a logical plan with the following logical operators:
-
Filteroperators with anAggregatechild operator -
Unary operators with the children resolved
ResolveSubquery is part of Resolution fixed-point batch of rules of the Spark Analyzer.
Technically, ResolveSubquery is a Catalyst rule for transforming logical plans, i.e. Rule[LogicalPlan].
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
// Use Catalyst DSL import org.apache.spark.sql.catalyst.expressions._ val a = 'a.int import org.apache.spark.sql.catalyst.plans.logical.LocalRelation val rel = LocalRelation(a) import org.apache.spark.sql.catalyst.expressions.Literal val list = Seq[Literal](1) // FIXME Use a correct query to demo ResolveSubquery import org.apache.spark.sql.catalyst.plans.logical.Filter import org.apache.spark.sql.catalyst.expressions.In val plan = Filter(condition = In(value = a, list), child = rel) scala> println(plan.numberedTreeString) 00 Filter a#9 IN (1) 01 +- LocalRelation <empty>, [a#9] import spark.sessionState.analyzer.ResolveSubquery val analyzedPlan = ResolveSubquery(plan) scala> println(analyzedPlan.numberedTreeString) 00 Filter a#9 IN (1) 01 +- LocalRelation <empty>, [a#9] |
Resolving Subquery Expressions (ScalarSubquery, Exists and In) — resolveSubQueries Internal Method
|
1 2 3 4 5 |
resolveSubQueries(plan: LogicalPlan, plans: Seq[LogicalPlan]): LogicalPlan |
resolveSubQueries requests the input logical plan to transform expressions (down the operator tree) as follows:
-
For ScalarSubquery expressions with subquery plan not resolved and resolveSubQuery to create resolved
ScalarSubqueryexpressions -
For Exists expressions with subquery plan not resolved and resolveSubQuery to create resolved
Existsexpressions -
For In expressions with ListQuery not resolved and resolveSubQuery to create resolved
Inexpressions
|
Note
|
resolveSubQueries is used exclusively when ResolveSubquery is executed.
|
resolveSubQuery Internal Method
|
1 2 3 4 5 6 7 8 |
resolveSubQuery( e: SubqueryExpression, plans: Seq[LogicalPlan])( f: (LogicalPlan, Seq[Expression]) => SubqueryExpression): SubqueryExpression |
resolveSubQuery…FIXME
|
Note
|
resolveSubQuery is used exclusively when ResolveSubquery is requested to resolve subquery expressions (ScalarSubquery, Exists and In).
|
Applying ResolveSubquery to Logical Plan (Executing ResolveSubquery) — apply Method
|
1 2 3 4 5 |
apply(plan: LogicalPlan): LogicalPlan |
|
Note
|
apply is part of Rule Contract to apply a rule to a TreeNode, e.g. logical plan.
|
apply transforms the input logical plan as follows:
-
For Filter operators with an Aggregate operator (as the child operator) and the children resolved,
applyresolves subquery expressions (ScalarSubquery, Exists and In) with theFilteroperator and the plans with theAggregateoperator and its single child -
For unary operators with the children resolved,
applyresolves subquery expressions (ScalarSubquery, Exists and In) with the unary operator and its single child
