spark技术分享Actions
Actions are RDD operations that produce non-RDD values. They materialize a value in a Spark program. In other words, a RDD operation that returns a value of any type but RDD[T] is an action.
|
1 2 3 4 5 |
action: RDD => a value |
|
Note
|
Actions are synchronous. You can use AsyncRDDActions to release a calling thread while calling actions. |
They trigger execution of RDD transformations to return values. Simply put, an action evaluates the RDD lineage graph.
You can think of actions as a valve and until action is fired, the data to be processed is not even in the pipes, i.e. transformations. Only actions can materialize the entire processing pipeline with real data.
Actions are one of two ways to send data from executors to the driver (the other being accumulators).
Actions in org.apache.spark.rdd.RDD:
-
aggregate -
collect -
count -
countApprox* -
countByValue* -
first -
fold -
foreach -
foreachPartition -
max -
min -
reduce -
saveAs* actions, e.g.
saveAsTextFile,saveAsHadoopFile -
take -
takeOrdered -
takeSample -
toLocalIterator -
top -
treeAggregate -
treeReduce
Actions run jobs using SparkContext.runJob or directly DAGScheduler.runJob.
|
1 2 3 4 5 6 |
scala> words.count (1) res0: Long = 502 |
-
wordsis an RDD ofString.
|
Tip
|
You should cache RDDs you work with when you want to execute two or more actions on it for a better performance. Refer to RDD Caching and Persistence. |
Before calling an action, Spark does closure/function cleaning (using SparkContext.clean) to make it ready for serialization and sending over the wire to executors. Cleaning can throw a SparkException if the computation cannot be cleaned.
|
Note
|
Spark uses ClosureCleaner to clean closures.
|
AsyncRDDActions
AsyncRDDActions class offers asynchronous actions that you can use on RDDs (thanks to the implicit conversion rddToAsyncRDDActions in RDD class). The methods return a FutureAction.
The following asynchronous methods are available:
-
countAsync -
collectAsync -
takeAsync -
foreachAsync -
foreachPartitionAsync
FutureActions
|
Caution
|
FIXME |
