spark技术分享Schedulable Builders
SchedulableBuilder is a contract of schedulable builders that operate on a pool of TaskSetManagers (from an owning TaskSchedulerImpl).
Schedulable builders can build pools and add new Schedulable entities to the pool.
|
Note
|
A SchedulableBuilder is created when TaskSchedulerImpl is being initialized. You can select the SchedulableBuilder to use by spark.scheduler.mode setting.
|
Spark comes with two implementations of the SchedulableBuilder Contract:
-
FIFOSchedulableBuilder – the default
SchedulableBuilder
|
Note
|
SchedulableBuilder is a private[spark] Scala trait. You can find the sources in org.apache.spark.scheduler.SchedulableBuilder.
|
SchedulableBuilder Contract
Every SchedulableBuilder provides the following services:
-
It manages a root pool.
-
It can build pools.
Root Pool (rootPool method)
|
1 2 3 4 5 |
rootPool: Pool |
rootPool method returns a Pool (of Schedulables).
This is the data structure managed (aka wrapped) by SchedulableBuilders.
Build Pools (buildPools method)
|
1 2 3 4 5 |
buildPools(): Unit |
|
Note
|
It is exclusively called by TaskSchedulerImpl.initialize. |
Adding Schedulable (to Pool) (addTaskSetManager method)
|
1 2 3 4 5 |
addTaskSetManager(manager: Schedulable, properties: Properties): Unit |
addTaskSetManager registers the manager Schedulable (with additional properties) to the rootPool.
|
Note
|
addTaskSetManager is exclusively used by TaskSchedulerImpl to submit a TaskSetManager for a stage for execution.
|
