ShuffleClient — Contract to Fetch Shuffle Blocks-spark技术分享

ShuffleClient — Contract to Fetch Shuffle Blocks

ShuffleClient is the contract of clients that can fetch shuffle block files.

ShuffleClient can optionally be initialized with an appId (that actually does nothing by default)

ShuffleClient has shuffle-related Spark metrics that are used when BlockManager is requested for a shuffle-related Spark metrics source (only when Executor is created for a non-local / cluster mode).



package org.apache.spark.network.shuffle;

abstract class ShuffleClient implements Closeable {
  // only required methods that have no implementation
  // the others follow
  abstract void fetchBlocks(
      String host,
      int port,
      String execId,
      String[] blockIds,
      BlockFetchingListener listener,
      TempFileManager tempFileManager);
}

package org.apache.spark.network.shuffle;

abstract class ShuffleClient implements Closeable {

// only required methods that have no implementation

// the others follow

abstract void fetchBlocks(

String host,

int port,

String execId,

String[] blockIds,

BlockFetchingListener listener,

TempFileManager tempFileManager);

}

Table 1. (Subset of) ShuffleClient Contract
Method	Description
`fetchBlocks`	Fetches a sequence of blocks from a remote block manager node asynchronously Used exclusively when `ShuffleBlockFetcherIterator` is requested to sendRequest

Table 2. ShuffleClients
ShuffleClient	Description
BlockTransferService
ExternalShuffleClient

`init` Method



void init(String appId)

void init(String appId)

init does nothing by default.

Note	`init` is used when: `BlockManager` is requested to initialize Spark on Mesos’ `MesosCoarseGrainedSchedulerBackend` is requested to `registered`

Requesting Shuffle-Related Spark Metrics — `shuffleMetrics` Method



MetricSet shuffleMetrics()

MetricSet shuffleMetrics()

shuffleMetrics returns an empty Dropwizard Metrics’ MetricSet by default.

Note	`shuffleMetrics` is used exclusively when `BlockManager` is requested for a shuffle-related Spark metrics source (only when `Executor` is created for a non-local / cluster mode).

ShuffleClient — Contract to Fetch Shuffle Blocks