spark技术分享StateStoreCoordinator — Tracking Locations of StateStores for StateStoreRDD
StateStoreCoordinator keeps track of StateStores loaded in Spark executors (across the nodes in a Spark cluster).
The main purpose of StateStoreCoordinator is for StateStoreRDD to get the location preferences for partitions (based on the location of the stores).
StateStoreCoordinator uses instances internal registry of StateStoreProviders by their ids and ExecutorCacheTaskLocations.
StateStoreCoordinator is a ThreadSafeRpcEndpoint RPC endpoint that manipulates instances registry through RPC messages.
| Message | Message Handler | ||
|---|---|---|---|
|
|
Removes You should see the following DEBUG message in the logs:
|
||
|
|
Gives the location of You should see the following DEBUG message in the logs:
|
||
|
|
|||
|
|
|||
|
|
Verifies if You should see the following DEBUG message in the logs:
|
|
Tip
|
Enable Add the following line to
Refer to Logging. |
