spark技术分享SortOrder Unevaluable Unary Expression
SortOrder is a unary expression that represents the following operators in a logical plan:
-
AstBuilderis requested to parse ORDER BY or SORT BY sort specifications -
Column.asc, Column.asc_nulls_first, Column.asc_nulls_last, Column.desc, Column.desc_nulls_first, and Column.desc_nulls_last operators are used
SortOrder is used to specify the output data ordering requirements of a physical operator.
SortOrder is an unevaluable expression and cannot be evaluated (i.e. produce a value given an internal row).
|
Note
|
An unevaluable expression cannot be evaluated to produce a value (neither in interpreted nor code-generated expression evaluations) and has to be resolved (replaced) to some other expressions or logical operators at analysis or optimization phases or they fail analysis. |
SortOrder is never foldable (as an unevaluable expression with no evaluation).
|
Tip
|
Use asc, asc_nullsLast, desc or desc_nullsFirst operators from the Catalyst DSL to create a SortOrder expression, e.g. for testing or Spark SQL internals exploration.
|
|
Note
|
Dataset.repartitionByRange, Dataset.sortWithinPartitions, Dataset.sort and WindowSpec.orderBy default to Ascending sort direction. |
Creating SortOrder Instance — apply Factory Method
|
1 2 3 4 5 6 7 8 |
apply( child: Expression, direction: SortDirection, sameOrderExpressions: Set[Expression] = Set.empty): SortOrder |
apply is a convenience method to create a SortOrder with the defaultNullOrdering of the SortDirection.
|
Note
|
apply is used exclusively in window function.
|
Catalyst DSL — asc, asc_nullsLast, desc and desc_nullsFirst Operators
|
1 2 3 4 5 6 7 8 |
asc: SortOrder asc_nullsLast: SortOrder desc: SortOrder desc_nullsFirst: SortOrder |
asc, asc_nullsLast, desc and desc_nullsFirst create a SortOrder expression with the Ascending or Descending sort direction, respectively.
|
1 2 3 4 5 6 7 8 |
import org.apache.spark.sql.catalyst.dsl.expressions._ val sortNullsLast = 'id.asc_nullsLast scala> println(sortNullsLast.sql) `id` ASC NULLS LAST |
Creating SortOrder Instance
SortOrder takes the following when created:
-
Child expression
-
“Same Order” expressions
SortDirection Contract
SortDirection is the base of sort directions.
| Method | Description | ||
|---|---|---|---|
|
|
Used when…FIXME |
||
|
|
Used when…FIXME |
Ascending and Descending Sort Directions
There are two sorting directions available, i.e. Ascending and Descending.
