spark技术分享LocalSchedulerBackend
LocalSchedulerBackend is a scheduler backend and a ExecutorBackend for Spark local run mode.
LocalSchedulerBackend acts as a “cluster manager” for local mode to offer resources on the single worker it manages, i.e. it calls TaskSchedulerImpl.resourceOffers(offers) with offers being a single-element collection with WorkerOffer.
|
Note
|
WorkerOffer represents a resource offer with CPU cores available on an executor. |
When an executor sends task status updates (using ExecutorBackend.statusUpdate), they are passed along as StatusUpdate to LocalEndpoint.
When LocalSchedulerBackend starts up, it registers a new RpcEndpoint called LocalSchedulerBackendEndpoint that is backed by LocalEndpoint. This is announced on LiveListenerBus as driver (using SparkListenerExecutorAdded message).
The application ids are in the format of local-[current time millis].
It communicates with LocalEndpoint using RPC messages.
The default parallelism is controlled using spark.default.parallelism property.
