spark技术分享MapOutputTrackerMasterEndpoint
MapOutputTrackerMasterEndpoint is a RpcEndpoint for MapOutputTrackerMaster.
MapOutputTrackerMasterEndpoint handles the following messages:
|
Tip
|
Enable Add the following line to
Refer to Logging. |
Creating MapOutputTrackerMasterEndpoint Instance
MapOutputTrackerMasterEndpoint takes the following when created:
-
rpcEnv— RpcEnv -
tracker— MapOutputTrackerMaster -
conf— SparkConf
When created, you should see the following DEBUG message in the logs:
|
1 2 3 4 5 |
DEBUG init |
|
Note
|
MapOutputTrackerMasterEndpoint is created when SparkEnv is created for the driver and executors.
|
GetMapOutputStatuses Message
|
1 2 3 4 5 6 |
GetMapOutputStatuses(shuffleId: Int) extends MapOutputTrackerMessage |
When GetMapOutputStatuses arrives, MapOutputTrackerMasterEndpoint reads the host and the port of the sender.
You should see the following INFO message in the logs:
|
1 2 3 4 5 |
INFO Asked to send map output locations for shuffle [shuffleId] to [hostPort] |
MapOutputTrackerMasterEndpoint posts a GetMapOutputMessage to MapOutputTrackerMaster (with shuffleId and the current RpcCallContext).
|
Note
|
GetMapOutputStatuses is posted when MapOutputTracker fetches shuffle map outputs remotely.
|
StopMapOutputTracker Message
|
1 2 3 4 5 6 |
StopMapOutputTracker extends MapOutputTrackerMessage |
When StopMapOutputTracker arrives, you should see the following INFO message in the logs:
|
1 2 3 4 5 |
INFO MapOutputTrackerMasterEndpoint stopped! |
MapOutputTrackerMasterEndpoint confirms the request (by replying true) and stops itself (and stops accepting messages).
|
Note
|
StopMapOutputTracker is posted when MapOutputTrackerMaster stops.
|
