spark技术分享StorageStatusListener — Spark Listener for Tracking BlockManagers
StorageStatusListener is a SparkListener that uses SparkListener callbacks to track status of every BlockManager in a Spark application.
StorageStatusListener is created and registered when SparkUI is created. It is later used to create ExecutorsListener and StorageListener Spark listeners.
| Callback | Description |
|---|---|
|
Adds an executor id with StorageStatus (with BlockManager and maximum memory on the executor) to executorIdToStorageStatus internal registry. Removes any other |
|
|
|
Removes an executor from executorIdToStorageStatus internal registry and adds the removed StorageStatus to deadExecutorStorageStatus internal registry. Removes the oldest StorageStatus when the number of entries in deadExecutorStorageStatus is bigger than spark.ui.retainedDeadExecutors. |
|
|
Updates StorageStatus for an executor in executorIdToStorageStatus internal registry, i.e. removes a block for |
|
Removes the RDD blocks for an unpersisted RDD (on every |
| Name | Description |
|---|---|
|
Collection of StorageStatus of removed/inactive Accessible using deadStorageStatusList method. Adds an element when Removes an element when |
|
|
Lookup table of StorageStatus per executor (including the driver). Adds an entry when Removes an entry when Updates |
Updating Storage Status For Executor — updateStorageStatus Method
|
Caution
|
FIXME |
Active BlockManagers (on Executors) — storageStatusList Method
|
1 2 3 4 5 |
storageStatusList: Seq[StorageStatus] |
storageStatusList gives a collection of StorageStatus (from executorIdToStorageStatus internal registry).
|
Note
|
|
deadStorageStatusList Method
|
1 2 3 4 5 |
deadStorageStatusList: Seq[StorageStatus] |
deadStorageStatusList gives deadExecutorStorageStatus internal registry.
|
Note
|
deadStorageStatusList is used when ExecutorsListener is requested for inactive/dead BlockManagers.
|
Removing RDD Blocks for Unpersisted RDD — updateStorageStatus Internal Method
|
1 2 3 4 5 |
updateStorageStatus(unpersistedRDDId: Int) |
updateStorageStatus takes active BlockManagers.
updateStorageStatus then finds RDD blocks for unpersistedRDDId RDD (for every BlockManager) and removes the blocks.
|
Note
|
storageStatusList is used exclusively when StorageStatusListener is notified that an RDD was unpersisted.
|
