WindowSpec — Window Specification-spark技术分享

WindowSpec — Window Specification

WindowSpec is a window specification that defines which rows are included in a window (frame), i.e. the set of rows that are associated with the current row by some relation.

WindowSpec takes the following when created:

Partition specification (Seq[Expression]) which defines which records are in the same partition. With no partition defined, all records belong to a single partition
Ordering Specification (Seq[SortOrder]) which defines how records in a partition are ordered that in turn defines the position of a record in a partition. The ordering could be ascending (ASC in SQL or asc in Scala) or descending (DESC or desc).
Frame Specification (WindowFrame) which defines the rows to be included in the frame for the current row, based on their relative position to the current row. For example, “the three rows preceding the current row to the current row” describes a frame including the current input row and three rows appearing before the current row.

You use Window object to create a WindowSpec.



import org.apache.spark.sql.expressions.Window
scala> val byHTokens = Window.partitionBy('token startsWith "h")
byHTokens: org.apache.spark.sql.expressions.WindowSpec = org.apache.spark.sql.expressions.WindowSpec@574985d8

import org.apache.spark.sql.expressions.Window

scala> val byHTokens = Window.partitionBy('token startsWith "h")

byHTokens: org.apache.spark.sql.expressions.WindowSpec = org.apache.spark.sql.expressions.WindowSpec@574985d8

Once the initial version of a WindowSpec is created, you use the methods to further configure the window specification.

Method Description

orderBy



orderBy(cols: Column*): WindowSpec
orderBy(colName: String, colNames: String*): WindowSpec

orderBy(cols: Column*): WindowSpec

orderBy(colName: String, colNames: String*): WindowSpec

partitionBy



partitionBy(cols: Column*): WindowSpec
partitionBy(colName: String, colNames: String*): WindowSpec

partitionBy(cols: Column*): WindowSpec

partitionBy(colName: String, colNames: String*): WindowSpec

rangeBetween



rangeBetween(start: Column, end: Column): WindowSpec
rangeBetween(start: Long, end: Long): WindowSpec

rangeBetween(start: Column, end: Column): WindowSpec

rangeBetween(start: Long, end: Long): WindowSpec

rowsBetween



rowsBetween(start: Long, end: Long): WindowSpec

rowsBetween(start: Long, end: Long): WindowSpec

With a window specification fully defined, you use Column.over operator that associates the WindowSpec with an aggregate or window function.



scala> :type windowSpec
org.apache.spark.sql.expressions.WindowSpec

import org.apache.spark.sql.functions.rank
val c = rank over windowSpec

scala> :type windowSpec

org.apache.spark.sql.expressions.WindowSpec

import org.apache.spark.sql.functions.rank

val c = rank over windowSpec

`withAggregate` Internal Method



withAggregate(aggregate: Column): Column

withAggregate(aggregate: Column): Column

withAggregate…FIXME

Note	`withAggregate` is used exclusively when Column.over operator is used.

WindowSpec — Window Specification