spark技术分享LocalEndpoint
LocalEndpoint is the communication channel between Task Scheduler and LocalSchedulerBackend. It is a (thread-safe) RpcEndpoint that hosts an executor (with id driver and hostname localhost) for Spark local mode.
When a LocalEndpoint starts up (as part of Spark local’s initialization) it prints out the following INFO messages to the logs:
|
1 2 3 4 5 6 |
INFO Executor: Starting executor ID driver on host localhost INFO Executor: Using REPL class URI: http://192.168.1.4:56131 |
reviveOffers Method
|
Caution
|
FIXME |
Creating LocalEndpoint Instance
|
Caution
|
FIXME |
RPC Messages
LocalEndpoint accepts the following RPC message types:
-
ReviveOffers(receive-only, non-blocking) – read Task Submission a.k.a. reviveOffers. -
StatusUpdate(receive-only, non-blocking) that passes the message to TaskScheduler (usingstatusUpdate) and if the task’s status is finished, it revives offers (seeReviveOffers). -
KillTask(receive-only, non-blocking) that kills the task that is currently running on the executor. -
StopExecutor(receive-reply, blocking) that stops the executor.
