spark技术分享 ResultStage — Final Stage in Job
A ResultStage is the final stage in a job that applies a function on one or many partitions of the target RDD to compute the result of an action.
Figure 1. Job creates ResultStage as the first stage
The partitions are given as a collection of partition ids (partitions) and the function func: (TaskContext, Iterator[_]) ⇒ _.
Figure 2.
ResultStage and partitions|
Tip
|
Read about TaskContext in TaskContext.
|
func Property
|
Caution
|
FIXME |
setActiveJob Method
|
Caution
|
FIXME |
removeActiveJob Method
|
Caution
|
FIXME |
activeJob Method
|
1 2 3 4 5 |
activeJob: Option[ActiveJob] |
activeJob returns the optional ActiveJob associated with a ResultStage.
|
Caution
|
FIXME When/why would that be NONE (empty)?
|
