BlockDataManager-spark技术分享

BlockDataManager — Block Storage Management API

BlockDataManager is the contract for managing storage for blocks of data (aka block storage management API).



package org.apache.spark.network

trait BlockDataManager {
  def getBlockData(blockId: BlockId): ManagedBuffer
  def putBlockData(
    blockId: BlockId,
    data: ManagedBuffer,
    level: StorageLevel,
    classTag: ClassTag[_]): Boolean
  def releaseLock(blockId: BlockId, taskAttemptId: Option[Long]): Unit
}

package org.apache.spark.network

trait BlockDataManager {

def getBlockData(blockId: BlockId): ManagedBuffer

def putBlockData(

blockId: BlockId,

data: ManagedBuffer,

level: StorageLevel,

classTag: ClassTag[_]): Boolean

def releaseLock(blockId: BlockId, taskAttemptId: Option[Long]): Unit

}

Note	`BlockDataManager` is a `private[spark]` contract.

Table 1. BlockDataManager Contract
Method	Description
`getBlockData`	Fetches a local block data by `blockId` Used when: `NettyBlockRpcServer` is requested to handle an OpenBlocks message `ShuffleBlockFetcherIterator` is requested to fetchLocalBlocks
`putBlockData`	Uploads a block data locally by `blockId`. The return value says whether the operation has succeeded (`true`) or failed (`false`). Used when…FIXME
`releaseLock`	Releases the lock for getBlockData and putBlockData methods Used when…FIXME

Blocks are identified by BlockId that has a globally unique identifier (name) and stored as ManagedBuffer.

Table 2. BlockIds
Name	Description
RDDBlockId	Described by RDD ID (`rddId`) and a partition index (`splitIndex`) Created when an `RDD` is requested to get or compute an RDD partition (identified by `splitIndex`).
`ShuffleBlockId`	Described by `shuffleId`, `mapId` and `reduceId`
ShuffleDataBlockId	Described by `shuffleId`, `mapId` and `reduceId`
ShuffleIndexBlockId	Described by `shuffleId`, `mapId` and `reduceId`
`BroadcastBlockId`	Described by `broadcastId` identifier and optional `field`
TaskResultBlockId	Described by `taskId`
StreamBlockId	Described by `streamId` and `uniqueId`

Note	BlockManager is the one and only known implementation of BlockDataManager Contract in Apache Spark.

BlockDataManager

BlockDataManager — Block Storage Management API

相关推荐

欢迎关注：spark技术分享

热门标签

近期文章

分类目录

关注公众号：spark技术分享

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏

QQ咨询

回顶部