Window Utility Object — Defining Window Specification-spark技术分享

Window Utility Object — Defining Window Specification

Window utility object is a set of static methods to define a window specification.

Method Description

currentRow



currentRow: Long

currentRow: Long

Value representing the current row that is used to define frame boundaries.

orderBy



orderBy(cols: Column*): WindowSpec
orderBy(colName: String, colNames: String*): WindowSpec

orderBy(cols: Column*): WindowSpec

orderBy(colName: String, colNames: String*): WindowSpec

Creates a WindowSpec with the ordering defined.

partitionBy



partitionBy(cols: Column*): WindowSpec
partitionBy(colName: String, colNames: String*): WindowSpec

partitionBy(cols: Column*): WindowSpec

partitionBy(colName: String, colNames: String*): WindowSpec

Creates a WindowSpec with the partitioning defined.

rangeBetween



rangeBetween(start: Column, end: Column): WindowSpec
rangeBetween(start: Long, end: Long): WindowSpec

rangeBetween(start: Column, end: Column): WindowSpec

rangeBetween(start: Long, end: Long): WindowSpec

Creates a WindowSpec with the frame boundaries defined, from start (inclusive) to end (inclusive). Both start and end are relative to the current row based on the actual value of the ORDER BY expression(s).

rowsBetween



rowsBetween(start: Long, end: Long): WindowSpec

rowsBetween(start: Long, end: Long): WindowSpec

Creates a WindowSpec with the frame boundaries defined, from start (inclusive) to end (inclusive). Both start and end are positions relative to the current row based on the position of the row within the partition.

unboundedFollowing



unboundedFollowing: Long

unboundedFollowing: Long

Value representing the last row in a partition (equivalent to “UNBOUNDED FOLLOWING” in SQL) that is used to define frame boundaries.

unboundedPreceding



unboundedPreceding: Long

unboundedPreceding: Long

Value representing the first row in a partition (equivalent to “UNBOUNDED PRECEDING” in SQL) that is used to define frame boundaries.



import org.apache.spark.sql.expressions.Window
import org.apache.spark.sql.functions.{currentRow, lit}
val windowSpec = Window
  .partitionBy($"orderId")
  .orderBy($"time")
  .rangeBetween(currentRow, lit(1))
scala> :type windowSpec
org.apache.spark.sql.expressions.WindowSpec

import org.apache.spark.sql.expressions.Window

import org.apache.spark.sql.functions.{currentRow, lit}

val windowSpec = Window

.partitionBy($"orderId")

.orderBy($"time")

.rangeBetween(currentRow, lit(1))

scala> :type windowSpec

org.apache.spark.sql.expressions.WindowSpec

Creating “Empty” WindowSpec — `spec` Internal Method



spec: WindowSpec

spec: WindowSpec

spec creates an “empty” WindowSpec, i.e. with empty partition and ordering specifications, and a UnspecifiedFrame.

Note	`spec` is used when: Column.over operator is used (with no `WindowSpec`) `Window` utility object is requested to partitionBy, orderBy, rowsBetween and rangeBetween

Window Utility Object — Defining Window Specification