spark技术分享AMEndpoint — ApplicationMaster RPC Endpoint
onStart Callback
When onStart is called, AMEndpoint communicates with the driver (the driver remote RPC Endpoint reference) by sending a one-way RegisterClusterManager message with a reference to itself.
After RegisterClusterManager has been sent (and received by YarnSchedulerEndpoint) the communication between the RPC endpoints of ApplicationMaster (YARN) and YarnSchedulerBackend (the Spark driver) is considered established.
RPC Messages
AddWebUIFilter
|
1 2 3 4 5 6 7 8 |
AddWebUIFilter( filterName: String, filterParams: Map[String, String], proxyBase: String) |
When AddWebUIFilter arrives, you should see the following INFO message in the logs:
|
1 2 3 4 5 |
INFO ApplicationMaster$AMEndpoint: Add WebUI Filter. [addWebUIFilter] |
It then passes the AddWebUIFilter message on to the driver’s scheduler backend (through YarnScheduler RPC Endpoint).
RequestExecutors
|
1 2 3 4 5 6 7 8 |
RequestExecutors( requestedTotal: Int, localityAwareTasks: Int, hostToLocalTaskCount: Map[String, Int]) |
When RequestExecutors arrives, AMEndpoint requests YarnAllocator for executors given locality preferences.
If the requestedTotal number of executors is different than the current number, resetAllocatorInterval is executed.
In case when YarnAllocator is not available yet, you should see the following WARN message in the logs:
|
1 2 3 4 5 |
WARN Container allocator is not ready to request executors yet. |
The response is false then.
resetAllocatorInterval
When RequestExecutors message arrives, it calls resetAllocatorInterval procedure.
|
1 2 3 4 5 |
resetAllocatorInterval(): Unit |
resetAllocatorInterval requests allocatorLock monitor lock and sets the internal nextAllocationInterval attribute to be initialAllocationInterval internal attribute. It then wakes up all threads waiting on allocatorLock.
|
Note
|
A thread waits on a monitor by calling one of the Object.wait methods.
|
