Column API — Column Operators-spark技术分享

Column API — Column Operators

Column API is a set of operators to work with values in a column (of a Dataset).

Operator Description

asc



asc: Column

asc: Column

asc_nulls_first



asc_nulls_first: Column

asc_nulls_first: Column

asc_nulls_last



asc_nulls_last: Column

asc_nulls_last: Column

desc



desc: Column

desc: Column

desc_nulls_first



desc_nulls_first: Column

desc_nulls_first: Column

desc_nulls_last



desc_nulls_last: Column

desc_nulls_last: Column

isin



isin(list: Any*): Column

isin(list: Any*): Column

isInCollection



isInCollection(values: scala.collection.Iterable[_]): Column

isInCollection(values: scala.collection.Iterable[_]): Column

(New in 2.4.0) An expression operator that is true if the value of the column is in the given values collection

isInCollection is simply a synonym of isin operator.

`isin` Operator



isin(list: Any*): Column

isin(list: Any*): Column

Internally, isin creates a Column with In predicate expression.



val ids = Seq((1, 2, 2), (2, 3, 1)).toDF("x", "y", "id")
scala> ids.show
+---+---+---+
|  x|  y| id|
+---+---+---+
|  1|  2|  2|
|  2|  3|  1|
+---+---+---+

val c = $"id" isin ($"x", $"y")
val q = ids.filter(c)
scala> q.show
+---+---+---+
|  x|  y| id|
+---+---+---+
|  1|  2|  2|
+---+---+---+

// Note that isin accepts non-Column values
val c = $"id" isin ("x", "y")
val q = ids.filter(c)
scala> q.show
+---+---+---+
|  x|  y| id|
+---+---+---+
+---+---+---+

val ids = Seq((1, 2, 2), (2, 3, 1)).toDF("x", "y", "id")

scala> ids.show

+---+---+---+

| x| y| id|

+---+---+---+

| 1| 2| 2|

| 2| 3| 1|

+---+---+---+

val c = $"id" isin ($"x", $"y")

val q = ids.filter(c)

scala> q.show

+---+---+---+

| x| y| id|

+---+---+---+

| 1| 2| 2|

+---+---+---+

// Note that isin accepts non-Column values

val c = $"id" isin ("x", "y")

val q = ids.filter(c)

scala> q.show

+---+---+---+

| x| y| id|

+---+---+---+

Column API — Column Operators