spark技术分享Schedulable
Schedulable is a contract of schedulable entities.
|
Note
|
Schedulable is a private[spark] Scala trait. You can find the sources in org.apache.spark.scheduler.Schedulable.
|
There are currently two types of Schedulable entities in Spark:
Schedulable Contract
Every Schedulable follows the following contract:
-
It has a
name.12345name: String -
It has a
parentPool (of otherSchedulables).12345parent: PoolWith the
parentproperty you could build a tree ofSchedulables -
It has a
schedulingMode,weight,minShare,runningTasks,priority,stageId.12345678910schedulingMode: SchedulingModeweight: IntminShare: IntrunningTasks: Intpriority: IntstageId: Int -
It manages a collection of Schedulables and can add or remove one.
1234567schedulableQueue: ConcurrentLinkedQueue[Schedulable]addSchedulable(schedulable: Schedulable): UnitremoveSchedulable(schedulable: Schedulable): UnitNoteschedulableQueueis java.util.concurrent.ConcurrentLinkedQueue. -
It can query for a
Schedulableby name.12345getSchedulableByName(name: String): Schedulable
-
It can be informed about lost executors.
12345executorLost(executorId: String, host: String, reason: ExecutorLossReason): UnitIt is called by TaskSchedulerImpl to inform TaskSetManagers about executors being lost.
-
It checks for speculatable tasks.
12345checkSpeculatableTasks(): BooleanCautionFIXME What are speculatable tasks?
getSortedTaskSetQueue
|
1 2 3 4 5 |
getSortedTaskSetQueue: ArrayBuffer[TaskSetManager] |
getSortedTaskSetQueue is used in TaskSchedulerImpl to handle resource offers (to let every TaskSetManager know about a new executor ready to execute tasks).
schedulableQueue
|
1 2 3 4 5 |
schedulableQueue: ConcurrentLinkedQueue[Schedulable] |
schedulableQueue is used in SparkContext.getAllPools.
