spark技术分享ResolveWindowFrame Logical Resolution Rule
ResolveWindowFrame is a logical resolution rule that the logical query plan analyzer uses to validate and resolve WindowExpression expressions in an entire logical query plan.
Technically, ResolveWindowFrame is just a Catalyst rule for transforming logical plans, i.e. Rule[LogicalPlan].
ResolveWindowFrame is part of Resolution fixed-point batch of rules.
ResolveWindowFrame takes a logical plan and does the following:
-
Makes sure that the window frame of a
WindowFunctionis unspecified or matches theSpecifiedWindowFrameof the WindowSpecDefinition expression.Reports a
AnalysisExceptionwhen the frames do not match:12345Window Frame [f] must match the required frame [frame] -
Copies the frame specification of
WindowFunctionto WindowSpecDefinition -
Creates a new
SpecifiedWindowFrameforWindowExpressionwith the resolved Catalyst expression andUnspecifiedFrame
|
Note
|
ResolveWindowFrame is a Scala object inside Analyzer class.
|
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
import import org.apache.spark.sql.expressions.Window // cume_dist requires ordered windows val q = spark. range(5). withColumn("cume_dist", cume_dist() over Window.orderBy("id")) import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan val planBefore: LogicalPlan = q.queryExecution.logical // Before ResolveWindowFrame scala> println(planBefore.numberedTreeString) 00 'Project [*, cume_dist() windowspecdefinition('id ASC NULLS FIRST, UnspecifiedFrame) AS cume_dist#39] 01 +- Range (0, 5, step=1, splits=Some(8)) import spark.sessionState.analyzer.ResolveWindowFrame val planAfter = ResolveWindowFrame.apply(plan) // After ResolveWindowFrame scala> println(planAfter.numberedTreeString) 00 'Project [*, cume_dist() windowspecdefinition('id ASC NULLS FIRST, RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS cume_dist#31] 01 +- Range (0, 5, step=1, splits=Some(8)) |
Applying ResolveWindowFrame to Logical Plan — apply Method
|
1 2 3 4 5 |
apply(plan: LogicalPlan): LogicalPlan |
|
Note
|
apply is part of Rule Contract to apply a rule to a logical plan.
|
apply…FIXME
