ShuffleReader-spark技术分享

ShuffleReader



package org.apache.spark.shuffle

trait ShuffleReader[K, C] {
  def read(): Iterator[Product2[K, C]]
}

package org.apache.spark.shuffle

trait ShuffleReader[K, C] {

def read(): Iterator[Product2[K, C]]

}

Note	`ShuffleReader` is a `private[spark]` contract.

Table 1. ShuffleReader Contract
Method	Description
`read`	Reading combined key-value records for a reduce task Used when: CoGroupedRDD, ShuffledRDD, and SubtractedRDD are requested to compute a partition (for a `ShuffleDependency` dependency) Spark SQL’s `ShuffledRowRDD` is requested to `compute` a partition

Note	BlockStoreShuffleReader is the one and only known ShuffleReader in Apache Spark.