spark技术分享WithWindowDefinition Unary Logical Operator
WithWindowDefinition is a unary logical plan with a single child logical plan and a windowDefinitions lookup table of WindowSpecDefinition per name.
WithWindowDefinition is created exclusively when AstBuilder parses window definitions.
The output schema of WithWindowDefinition is exactly the output attributes of the child logical operator.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
// Example with window specification alias and definition val sqlText = """ SELECT count(*) OVER anotherWindowSpec FROM range(5) WINDOW anotherWindowSpec AS myWindowSpec, myWindowSpec AS ( PARTITION BY id RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ) """ import spark.sessionState.{analyzer, sqlParser} val parsedPlan = sqlParser.parsePlan(sqlText) scala> println(parsedPlan.numberedTreeString) 00 'WithWindowDefinition Map(anotherWindowSpec -> windowspecdefinition('id, RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW), myWindowSpec -> windowspecdefinition('id, RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)) 01 +- 'Project [unresolvedalias(unresolvedwindowexpression('count(1), WindowSpecReference(anotherWindowSpec)), None)] 02 +- 'UnresolvedTableValuedFunction range, [5] val plan = analyzer.execute(parsedPlan) scala> println(plan.numberedTreeString) 00 Project [count(1) OVER (PARTITION BY id RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)#75L] 01 +- Project [id#73L, count(1) OVER (PARTITION BY id RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)#75L, count(1) OVER (PARTITION BY id RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)#75L] 02 +- Window [count(1) windowspecdefinition(id#73L, RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS count(1) OVER (PARTITION BY id RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)#75L], [id#73L] 03 +- Project [id#73L] 04 +- Range (0, 5, step=1, splits=None) |
