spark技术分享StorageStatus
StorageStatus is a developer API that Spark uses to pass “just enough” information about registered BlockManagers in a Spark application between Spark services (mostly for monitoring purposes like web UI or SparkListeners).
|
Note
|
There are two ways to access
|
StorageStatus is created when:
-
BlockManagerMasterEndpointis requested for storage status (of every BlockManager in a Spark application) -
StorageStatusListenergets notified about a newBlockManager(in a Spark application)
| Name | Description |
|---|---|
|
Lookup table of Used when…FIXME |
|
|
Lookup table of Used when…FIXME |
updateStorageInfo Method
|
Caution
|
FIXME |
Creating StorageStatus Instance
StorageStatus takes the following when created:
StorageStatus initializes the internal registries and counters.
Getting RDD Blocks For RDD — rddBlocksById Method
|
1 2 3 4 5 |
rddBlocksById(rddId: Int): Map[BlockId, BlockStatus] |
rddBlocksById gives the blocks (as BlockId with their status as BlockStatus) that belong to rddId RDD.
|
Note
|
|
Removing Block (From Internal Registries) — removeBlock Internal Method
|
1 2 3 4 5 |
removeBlock(blockId: BlockId): Option[BlockStatus] |
removeBlock removes blockId from _rddBlocks registry and returns it.
Internally, removeBlock updates block status of blockId (to be empty, i.e. removed).
removeBlock branches off per the type of BlockId, i.e. RDDBlockId or not.
For a RDDBlockId, removeBlock finds the RDD in _rddBlocks and removes the blockId. removeBlock removes the RDD (from _rddBlocks) completely, if there are no more blocks registered.
For a non-RDDBlockId, removeBlock removes blockId from _nonRddBlocks registry.
|
Note
|
removeBlock is used when StorageStatusListener removes RDD blocks for an unpersisted RDD or updates storage status for an executor.
|
