spark技术分享JobWaiter
|
1 2 3 4 5 6 7 8 9 10 |
JobWaiter[T]( dagScheduler: DAGScheduler, val jobId: Int, totalTasks: Int, resultHandler: (Int, T) => Unit) extends JobListener |
JobWaiter is a JobListener that is used when DAGScheduler submits a job or submits a map stage.
You can use a JobWaiter to block until the job finishes executing or to cancel it.
While the methods execute, JobSubmitted and MapStageSubmitted events are posted that reference the JobWaiter.
As a JobListener, JobWaiter gets notified about task completions or failures, using taskSucceeded and jobFailed, respectively. When the total number of tasks (that equals the number of partitions to compute) equals the number of taskSucceeded, the JobWaiter instance is marked successful. A jobFailed event marks the JobWaiter instance failed.
