spark技术分享WindowExpression Unevaluable Expression
WindowExpression is an unevaluable expression that represents a window function (over some WindowSpecDefinition).
|
Note
|
An unevaluable expression cannot be evaluated to produce a value (neither in interpreted nor code-generated expression evaluations) and has to be resolved (replaced) to some other expressions or logical operators at analysis or optimization phases or they fail analysis. |
WindowExpression is created when:
-
WindowSpecis requested to withAggregate (when Column.over operator is used) -
WindowsSubstitutionlogical evaluation rule is executed (with WithWindowDefinition logical operators with UnresolvedWindowExpression expressions) -
AstBuilderis requested to parse a function call in a SQL statement
|
Note
|
WindowExpression can only be created with AggregateExpression, AggregateWindowFunction or OffsetWindowFunction expressions which is enforced at analysis.
|
|
1 2 3 4 5 6 7 |
// Using Catalyst DSL val wf = 'count.function(star()) val windowSpec = ??? |
|
Note
|
WindowExpression is resolved in ExtractWindowExpressions, ResolveWindowFrame and ResolveWindowOrder logical rules.
|
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
import org.apache.spark.sql.catalyst.expressions.WindowExpression // relation - Dataset as a table to query val table = spark.emptyDataset[Int] scala> val windowExpr = table .selectExpr("count() OVER (PARTITION BY value) AS count") .queryExecution .logical (1) .expressions .toList(0) .children(0) .asInstanceOf[WindowExpression] windowExpr: org.apache.spark.sql.catalyst.expressions.WindowExpression = 'count() windowspecdefinition('value, UnspecifiedFrame) scala> windowExpr.sql res2: String = count() OVER (PARTITION BY `value` UnspecifiedFrame) |
-
Use
sqlParserdirectly as in WithWindowDefinition Example
| Name | Description |
|---|---|
|
|
Collection of two expressions, i.e. windowFunction and WindowSpecDefinition, for which |
|
|
|
|
|
Whether or not windowFunction is foldable. |
|
|
Whether or not windowFunction is nullable. |
|
|
|
|
|
|
|
Note
|
WindowExpression is subject to NullPropagation and DecimalAggregates logical optimizations.
|
|
Note
|
Distinct window functions are not supported which is enforced at analysis. |
|
Note
|
An offset window function can only be evaluated in an ordered row-based window frame with a single offset which is enforced at analysis. |
Catalyst DSL — windowExpr Operator
|
1 2 3 4 5 |
windowExpr(windowFunc: Expression, windowSpec: WindowSpecDefinition): WindowExpression |
windowExpr operator in Catalyst DSL creates a WindowExpression expression, e.g. for testing or Spark SQL internals exploration.
|
1 2 3 4 5 |
// FIXME: DEMO |
Creating WindowExpression Instance
WindowExpression takes the following when created:
-
Window function expression
-
WindowSpecDefinition expression
