ShuffleWriter-spark技术分享

ShuffleWriter

Caution



abstract class ShuffleWriter[K, V] {
  def write(records: Iterator[Product2[K, V]]): Unit
  def stop(success: Boolean): Option[MapStatus]
}

abstract class ShuffleWriter[K, V] {

def write(records: Iterator[Product2[K, V]]): Unit

def stop(success: Boolean): Option[MapStatus]

}

Note	`ShuffleWriter` is a `private[spark]` contract.

Table 1. `ShuffleWriter` Contract
Method	Description
`write`	Writes a sequence of records (for a RDD partition) to a shuffle system when a `ShuffleMapTask` writes its execution result.
`stop`	Closes a `ShuffleWriter` and returns MapStatus if the writing completed successfully. Used when a `ShuffleMapTask` finishes execution with the input `success` flag to match the status of the task execution.