spark技术分享Stages for All Jobs Page
AllStagesPage is a web page (section) that is registered with the Stages tab that displays all stages in a Spark application – active, pending, completed, and failed stages with their count.
In FAIR scheduling mode you have access to the table showing the scheduler pools as well as the pool names per stage.
|
Note
|
Pool names are calculated using SparkContext.getAllPools. |
Internally, AllStagesPage is a WebUIPage with access to the parent Stages tab and more importantly the JobProgressListener to have access to current state of the entire Spark application.
Rendering AllStagesPage (render method)
|
1 2 3 4 5 |
render(request: HttpServletRequest): Seq[Node] |
render generates a HTML page to display in a web browser.
It uses the parent’s JobProgressListener to know about:
-
active stages (as
activeStages) -
pending stages (as
pendingStages) -
completed stages (as
completedStages) -
failed stages (as
failedStages) -
the number of completed stages (as
numCompletedStages) -
the number of failed stages (as
numFailedStages)
|
Note
|
Stage information is available as StageInfo object. |
There are 4 different tables for the different states of stages – active, pending, completed, and failed. They are displayed only when there are stages in a given state.
You could also notice “retry” for stage when it was retried.
|
Caution
|
FIXME A screenshot |
