BlockTransferService — Pluggable Block Transfers (To Fetch and Upload Blocks)-spark技术分享

BlockTransferService — Pluggable Block Transfers (To Fetch and Upload Blocks)

BlockTransferService is the base for ShuffleClients that can fetch and upload blocks of data synchronously or asynchronously.



package org.apache.spark.network

abstract class BlockTransferService extends ShuffleClient {
  // only required methods that have no implementation
  // the others follow
  def init(blockDataManager: BlockDataManager): Unit
  def close(): Unit
  def port: Int
  def hostName: String
  def fetchBlocks(
    host: String,
    port: Int,
    execId: String,
    blockIds: Array[String],
    listener: BlockFetchingListener,
    tempFileManager: TempFileManager): Unit
  def uploadBlock(
    hostname: String,
    port: Int,
    execId: String,
    blockId: BlockId,
    blockData: ManagedBuffer,
    level: StorageLevel,
    classTag: ClassTag[_]): Future[Unit]
}

package org.apache.spark.network

abstract class BlockTransferService extends ShuffleClient {

// only required methods that have no implementation

// the others follow

def init(blockDataManager: BlockDataManager): Unit

def close(): Unit

def port: Int

def hostName: String

def fetchBlocks(

host: String,

port: Int,

execId: String,

blockIds: Array[String],

listener: BlockFetchingListener,

tempFileManager: TempFileManager): Unit

def uploadBlock(

hostname: String,

port: Int,

execId: String,

blockId: BlockId,

blockData: ManagedBuffer,

level: StorageLevel,

classTag: ClassTag[_]): Future[Unit]

}

Note	`BlockTransferService` is a `private[spark]` contract.

Method Description

init

Used when…FIXME

close

Used when…FIXME

port

Used when…FIXME

hostName

Used when…FIXME

fetchBlocks

Fetches a sequence of blocks from a remote node asynchronously

Used exclusively when BlockTransferService is requested to fetch only one block (in a blocking fashion)

Note	`fetchBlocks` is part of ShuffleClient Contract to…FIXME.

uploadBlock

Used exclusively when BlockTransferService is requested to upload a single block to a remote node (in a blocking fashion).

Note	NettyBlockTransferService is the one and only known implementation of BlockTransferService Contract.

Note	`BlockTransferService` was introduced in SPARK-3019 Pluggable block transfer interface (BlockTransferService) and is available since Spark 1.2.0.

`fetchBlockSync` Method



fetchBlockSync(
  host: String,
  port: Int,
  execId: String,
  blockId: String,
  tempFileManager: TempFileManager): ManagedBuffer

fetchBlockSync(

host: String,

port: Int,

execId: String,

blockId: String,

tempFileManager: TempFileManager): ManagedBuffer

fetchBlockSync…FIXME

Synchronous (and hence blocking) fetchBlockSync to fetch one block blockId (that corresponds to the ShuffleClient parent’s asynchronous fetchBlocks).

fetchBlockSync is a mere wrapper around fetchBlocks to fetch one blockId block that waits until the fetch finishes.

Note	`fetchBlockSync` is used when…FIXME

Uploading Single Block to Remote Node (Blocking Fashion) — `uploadBlockSync` Method



uploadBlockSync(
  hostname: String,
  port: Int,
  execId: String,
  blockId: BlockId,
  blockData: ManagedBuffer,
  level: StorageLevel,
  classTag: ClassTag[_]): Unit

uploadBlockSync(

hostname: String,

port: Int,

execId: String,

blockId: BlockId,

blockData: ManagedBuffer,

level: StorageLevel,

classTag: ClassTag[_]): Unit

uploadBlockSync…FIXME

uploadBlockSync is a mere blocking wrapper around uploadBlock that waits until the upload finishes.

Note	`uploadBlockSync` is used exclusively when `BlockManager` is requested to replicate (when a replication level is greater than 1).

BlockTransferService — Pluggable Block Transfers (To Fetch and Upload Blocks)