Actions-spark技术分享

Dataset API — Actions

Actions are part of the Dataset API for…FIXME

Note	Actions are the methods in the `Dataset` Scala class that are grouped in `action` group name, i.e. `@group action`.

Action Description

collect



collect(): Array[T]

collect(): Array[T]

count



count(): Long

count(): Long

describe



describe(cols: String*): DataFrame

describe(cols: String*): DataFrame

first



first(): T

first(): T

foreach



foreach(f: T => Unit): Unit

foreach(f: T => Unit): Unit

foreachPartition



foreachPartition(f: Iterator[T] => Unit): Unit

foreachPartition(f: Iterator[T] => Unit): Unit

head



head(): T
head(n: Int): Array[T]

head(): T

head(n: Int): Array[T]

reduce



reduce(func: (T, T) => T): T

reduce(func: (T, T) => T): T

show



show(): Unit
show(truncate: Boolean): Unit
show(numRows: Int): Unit
show(numRows: Int, truncate: Boolean): Unit
show(numRows: Int, truncate: Int): Unit
show(numRows: Int, truncate: Int, vertical: Boolean): Unit

show(): Unit

show(truncate: Boolean): Unit

show(numRows: Int): Unit

show(numRows: Int, truncate: Boolean): Unit

show(numRows: Int, truncate: Int): Unit

show(numRows: Int, truncate: Int, vertical: Boolean): Unit

summary

Computes specified statistics for numeric and string columns. The default statistics are: count, mean, stddev, min, max and 25%, 50%, 75% percentiles.



summary(statistics: String*): DataFrame

summary(statistics: String*): DataFrame

Note	`summary` is an extended version of the describe action that simply calculates `count`, `mean`, `stddev`, `min` and `max` statistics.

take



take(n: Int): Array[T]

take(n: Int): Array[T]

toLocalIterator



toLocalIterator(): java.util.Iterator[T]

toLocalIterator(): java.util.Iterator[T]

`collect` Action



collect(): Array[T]

collect(): Array[T]

collect…FIXME

`count` Action



count(): Long

count(): Long

count…FIXME

Calculating Basic Statistics — `describe` Action



describe(cols: String*): DataFrame

describe(cols: String*): DataFrame

describe…FIXME

`first` Action



first(): T

first(): T

first…FIXME

`foreach` Action



foreach(f: T => Unit): Unit

foreach(f: T => Unit): Unit

foreach…FIXME

`foreachPartition` Action



foreachPartition(f: Iterator[T] => Unit): Unit

foreachPartition(f: Iterator[T] => Unit): Unit

foreachPartition…FIXME

`head` Action



head(): T (1)
head(n: Int): Array[T]

head(): T (1)

head(n: Int): Array[T]

Calls the other head with n as 1 and takes the first element

head…FIXME

`reduce` Action



reduce(func: (T, T) => T): T

reduce(func: (T, T) => T): T

reduce…FIXME

`show` Action



show(): Unit
show(truncate: Boolean): Unit
show(numRows: Int): Unit
show(numRows: Int, truncate: Boolean): Unit
show(numRows: Int, truncate: Int): Unit
show(numRows: Int, truncate: Int, vertical: Boolean): Unit

show(): Unit

show(truncate: Boolean): Unit

show(numRows: Int): Unit

show(numRows: Int, truncate: Boolean): Unit

show(numRows: Int, truncate: Int): Unit

show(numRows: Int, truncate: Int, vertical: Boolean): Unit

show…FIXME

Calculating Statistics — `summary` Action



summary(statistics: String*): DataFrame

summary(statistics: String*): DataFrame

summary calculates specified statistics for numeric and string columns.

The default statistics are: count, mean, stddev, min, max and 25%, 50%, 75% percentiles.

Note	`summary` accepts arbitrary approximate percentiles specified as a percentage (e.g. `10%`).

Internally, summary uses the StatFunctions to calculate the requested summaries for the Dataset.

Taking First Records — `take` Action



take(n: Int): Array[T]

take(n: Int): Array[T]

take is an action on a Dataset that returns a collection of n records.

Warning

take loads all the data into the memory of the Spark application’s driver process and for a large n could result in OutOfMemoryError.

Internally, take creates a new Dataset with Limit logical plan for Literal expression and the current LogicalPlan. It then runs the SparkPlan that produces a Array[InternalRow] that is in turn decoded to Array[T] using a bounded encoder.

`toLocalIterator` Action



toLocalIterator(): java.util.Iterator[T]

toLocalIterator(): java.util.Iterator[T]

toLocalIterator…FIXME

Actions

Dataset API — Actions

`collect` Action

`count` Action

Calculating Basic Statistics — `describe` Action

`first` Action

`foreach` Action

`foreachPartition` Action

`head` Action

`reduce` Action

`show` Action

Calculating Statistics — `summary` Action

Taking First Records — `take` Action

`toLocalIterator` Action

相关推荐

欢迎关注：spark技术分享

热门标签

近期文章

分类目录

关注公众号：spark技术分享

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏

QQ咨询

回顶部

Dataset API — Actions

collect Action

count Action

Calculating Basic Statistics — describe Action

first Action

foreach Action

foreachPartition Action

head Action

reduce Action

show Action

Calculating Statistics — summary Action

Taking First Records — take Action

toLocalIterator Action

相关推荐

欢迎关注：spark技术分享

热门标签

近期文章

分类目录

关注公众号：spark技术分享

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏

QQ咨询

回顶部

`collect` Action

`count` Action

Calculating Basic Statistics — `describe` Action

`first` Action

`foreach` Action

`foreachPartition` Action

`head` Action

`reduce` Action

`show` Action

Calculating Statistics — `summary` Action

Taking First Records — `take` Action

`toLocalIterator` Action