ResolveSubquery Logical Resolution Rule
ResolveSubquery
is a logical resolution that resolves subquery expressions (ScalarSubquery, Exists and In) when transforming a logical plan with the following logical operators:
-
Filter
operators with anAggregate
child operator -
Unary operators with the children resolved
ResolveSubquery
is part of Resolution fixed-point batch of rules of the Spark Analyzer.
Technically, ResolveSubquery
is a Catalyst rule for transforming logical plans, i.e. Rule[LogicalPlan]
.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
// Use Catalyst DSL import org.apache.spark.sql.catalyst.expressions._ val a = 'a.int import org.apache.spark.sql.catalyst.plans.logical.LocalRelation val rel = LocalRelation(a) import org.apache.spark.sql.catalyst.expressions.Literal val list = Seq[Literal](1) // FIXME Use a correct query to demo ResolveSubquery import org.apache.spark.sql.catalyst.plans.logical.Filter import org.apache.spark.sql.catalyst.expressions.In val plan = Filter(condition = In(value = a, list), child = rel) scala> println(plan.numberedTreeString) 00 Filter a#9 IN (1) 01 +- LocalRelation <empty>, [a#9] import spark.sessionState.analyzer.ResolveSubquery val analyzedPlan = ResolveSubquery(plan) scala> println(analyzedPlan.numberedTreeString) 00 Filter a#9 IN (1) 01 +- LocalRelation <empty>, [a#9] |
Resolving Subquery Expressions (ScalarSubquery, Exists and In) — resolveSubQueries
Internal Method
1 2 3 4 5 |
resolveSubQueries(plan: LogicalPlan, plans: Seq[LogicalPlan]): LogicalPlan |
resolveSubQueries
requests the input logical plan to transform expressions (down the operator tree) as follows:
-
For ScalarSubquery expressions with subquery plan not resolved and resolveSubQuery to create resolved
ScalarSubquery
expressions -
For Exists expressions with subquery plan not resolved and resolveSubQuery to create resolved
Exists
expressions -
For In expressions with ListQuery not resolved and resolveSubQuery to create resolved
In
expressions
Note
|
resolveSubQueries is used exclusively when ResolveSubquery is executed.
|
resolveSubQuery
Internal Method
1 2 3 4 5 6 7 8 |
resolveSubQuery( e: SubqueryExpression, plans: Seq[LogicalPlan])( f: (LogicalPlan, Seq[Expression]) => SubqueryExpression): SubqueryExpression |
resolveSubQuery
…FIXME
Note
|
resolveSubQuery is used exclusively when ResolveSubquery is requested to resolve subquery expressions (ScalarSubquery, Exists and In).
|
Applying ResolveSubquery to Logical Plan (Executing ResolveSubquery) — apply
Method
1 2 3 4 5 |
apply(plan: LogicalPlan): LogicalPlan |
Note
|
apply is part of Rule Contract to apply a rule to a TreeNode, e.g. logical plan.
|
apply
transforms the input logical plan as follows:
-
For Filter operators with an Aggregate operator (as the child operator) and the children resolved,
apply
resolves subquery expressions (ScalarSubquery, Exists and In) with theFilter
operator and the plans with theAggregate
operator and its single child -
For unary operators with the children resolved,
apply
resolves subquery expressions (ScalarSubquery, Exists and In) with the unary operator and its single child