StagesTab — Stages for All Jobs-spark技术分享

StagesTab — Stages for All Jobs

StagesTab is a SparkUITab with stages prefix.

StagesTab is created exclusively when SparkUI is initialized.

When created, StagesTab creates the following pages and attaches them immediately:

Stages tab in web UI shows the current state of all stages of all jobs in a Spark application (i.e. a SparkContext) with two optional pages for the tasks and statistics for a stage (when a stage is selected) and pool details (when the application works in FAIR scheduling mode).

The title of the tab is Stages for All Jobs.

You can access the Stages tab under /stages URL, i.e. http://localhost:4040/stages.

With no jobs submitted yet (and hence no stages to display), the page shows nothing but the title.

Figure 1. Stages Page Empty

The Stages page shows the stages in a Spark application per state in their respective sections — Active Stages, Pending Stages, Completed Stages, and Failed Stages.

Figure 2. Stages Page With One Stage Completed

Note	The state sections are only displayed when there are stages in a given state. Refer to Stages for All Jobs.

In FAIR scheduling mode you have access to the table showing the scheduler pools.

spark webui stages fairschedulerpools.png

Figure 3. Fair Scheduler Pools Table

Internally, the page is represented by org.apache.spark.ui.jobs.StagesTab class.

The page uses the parent’s SparkUI to access required services, i.e. SparkContext, SparkConf, JobProgressListener, RDDOperationGraphListener, and to know whether kill is enabled or not.

StagesTab is created when…FIXME

`killEnabled` flag

Caution

FIXME

Creating StagesTab Instance

StagesTab takes the following when created:

Handling Request to Kill Stage (from web UI) — `handleKillRequest` Method



handleKillRequest(request: HttpServletRequest): Unit

handleKillRequest(request: HttpServletRequest): Unit

handleKillRequest…FIXME

Note	`handleKillRequest` is used when…FIXME

StagesTab — Stages for All Jobs