spark技术分享ExtractWindowExpressions Logical Resolution Rule
ExtractWindowExpressions is a logical resolution rule that transforms a logical query plan and replaces (extracts) WindowExpression expressions with Window logical operators.
ExtractWindowExpressions is part of the Resolution fixed-point batch in the standard batches of the Analyzer.
ExtractWindowExpressions is simply a Catalyst rule for transforming logical plans, i.e. Rule[LogicalPlan].
|
Note
|
ExtractWindowExpressions is a Scala object inside Analyzer class (so you have to create an instance of the Analyzer class to access it or simply use SessionState).
|
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
import spark.sessionState.analyzer.ExtractWindowExpressions // Example 1: Filter + Aggregate with WindowExpressions in aggregateExprs val q = ??? val plan = q.queryExecution.logical val afterExtractWindowExpressions = ExtractWindowExpressions(plan) // Example 2: Aggregate with WindowExpressions in aggregateExprs val q = ??? val plan = q.queryExecution.logical val afterExtractWindowExpressions = ExtractWindowExpressions(plan) // Example 3: Project with WindowExpressions in projectList val q = ??? val plan = q.queryExecution.logical val afterExtractWindowExpressions = ExtractWindowExpressions(plan) |
Executing Rule — apply Method
|
1 2 3 4 5 |
apply(plan: LogicalPlan): LogicalPlan |
|
Note
|
apply is part of the Rule Contract to execute (apply) a rule on a TreeNode (e.g. LogicalPlan).
|
apply transforms the logical operators downwards in the input logical plan as follows:
-
For Filter unary operators with Aggregate operator (as the child) that has a window function in the aggregateExpressions,
apply…FIXME -
For Aggregate logical operators that have a window function in the aggregateExpressions,
apply…FIXME -
For Project logical operators that have a window function in the projectList,
apply…FIXME
hasWindowFunction Internal Method
|
1 2 3 4 5 6 |
hasWindowFunction(projectList: Seq[NamedExpression]): Boolean (1) hasWindowFunction(expr: NamedExpression): Boolean |
-
Executes the other
hasWindowFunctionon everyNamedExpressionin theprojectList
hasWindowFunction is positive (true) when the input expr named expression is a WindowExpression expression. Otherwise, hasWindowFunction is negative (false).
extract Internal Method
|
1 2 3 4 5 |
extract(expressions: Seq[NamedExpression]): (Seq[NamedExpression], Seq[NamedExpression]) |
extract…FIXME
|
Note
|
extract is used exclusively when ExtractWindowExpressions logical resolution rule is executed.
|
Adding Project and Window Logical Operators to Logical Plan — addWindow Internal Method
|
1 2 3 4 5 6 7 |
addWindow( expressionsWithWindowFunctions: Seq[NamedExpression], child: LogicalPlan): LogicalPlan |
addWindow adds a Project logical operator with one or more Window logical operators (for every WindowExpression in the input named expressions) to the input logical plan.
Internally, addWindow…FIXME
|
Note
|
addWindow is used exclusively when ExtractWindowExpressions logical resolution rule is executed.
|
