SparkUI — Web UI of Spark Application-spark技术分享

SparkUI — Web UI of Spark Application

SparkUI is the web UI of a Spark application (aka Application UI).

SparkUI is created along with the following:

SparkContext (for a live Spark application with spark.ui.enabled configuration property enabled)
FsHistoryProvider is requested for the application UI (for a live or completed Spark application)

Figure 1. Creating SparkUI for Live Spark Application

When created (while SparkContext is created for a live Spark application), SparkUI gets the following:

Live AppStatusStore (with a ElementTrackingStore using an InMemoryStore and a live AppStatusListener)
Name of the Spark application that is exactly the value of spark.app.name configuration property
Empty base path

When started, SparkUI binds to appUIAddress address that you can control using SPARK_PUBLIC_DNS environment variable or spark.driver.host Spark property.

Note	With spark.ui.killEnabled configuration property turned on, `SparkUI` allows to kill jobs and stages (subject to `SecurityManager.checkModifyPermissions` permissions).

SparkUI gets an AppStatusStore that is then used for the following:

Initializing tabs, i.e. JobsTab, StagesTab, StorageTab, EnvironmentTab
AbstractApplicationResource is requested for jobsList, oneJob, executorList, allExecutorList, rddList, rddData, environmentInfo
StagesResource is requested for stageList, stageData, oneAttemptData, taskSummary, taskList
SparkUI is requested for the current Spark user
Creating Spark SQL’s SQLTab (when SQLHistoryServerPlugin is requested to setupUI)
Spark Streaming’s BatchPage is created

Table 1. SparkUI’s Internal Properties (e.g. Registries, Counters and Flags)
Name	Description
`appId`

Tip

Enable INFO logging level for org.apache.spark.ui.SparkUI logger to see what happens inside.

Add the following line to conf/log4j.properties:



log4j.logger.org.apache.spark.ui.SparkUI=INFO

log4j.logger.org.apache.spark.ui.SparkUI=INFO

Refer to Logging.

Assigning Unique Identifier of Spark Application — `setAppId` Method



setAppId(id: String): Unit

setAppId(id: String): Unit

setAppId sets the internal appId.

Note	`setAppId` is used exclusively when `SparkContext` is initialized.

Stopping `SparkUI` — `stop` Method



stop(): Unit

stop(): Unit

stop stops the HTTP server and prints the following INFO message to the logs:



INFO SparkUI: Stopped Spark web UI at [appUIAddress]

INFO SparkUI: Stopped Spark web UI at [appUIAddress]

Note	`appUIAddress` in the above INFO message is the result of appUIAddress method.

`appUIAddress` Method



appUIAddress: String

appUIAddress: String

appUIAddress returns the entire URL of a Spark application’s web UI, including http:// scheme.

Internally, appUIAddress uses appUIHostPort.

Accessing Spark User — `getSparkUser` Method



getSparkUser: String

getSparkUser: String

getSparkUser returns the name of the user a Spark application runs as.

Internally, getSparkUser requests user.name System property from EnvironmentListener Spark listener.

Note	`getSparkUser` is used…FIXME

`createLiveUI` Method



createLiveUI(
  sc: SparkContext,
  conf: SparkConf,
  listenerBus: SparkListenerBus,
  jobProgressListener: JobProgressListener,
  securityManager: SecurityManager,
  appName: String,
  startTime: Long): SparkUI

createLiveUI(

sc: SparkContext,

conf: SparkConf,

listenerBus: SparkListenerBus,

jobProgressListener: JobProgressListener,

securityManager: SecurityManager,

appName: String,

startTime: Long): SparkUI

createLiveUI creates a SparkUI for a live running Spark application.

Internally, createLiveUI simply forwards the call to create.

Note	`createLiveUI` is called when `SparkContext` is created (and spark.ui.enabled is enabled).

`createHistoryUI` Method

Caution

FIXME

`appUIHostPort` Method



appUIHostPort: String

appUIHostPort: String

appUIHostPort returns the Spark application’s web UI which is the public hostname and port, excluding the scheme.

Note	appUIAddress uses `appUIHostPort` and adds `http://` scheme.

`getAppName` Method



getAppName: String

getAppName: String

getAppName returns the name of the Spark application (of a SparkUI instance).

Note	`getAppName` is used when…FIXME

Creating SparkUI Instance — `create` Factory Method



create(
  sc: Option[SparkContext],
  store: AppStatusStore,
  conf: SparkConf,
  securityManager: SecurityManager,
  appName: String,
  basePath: String = "",
  startTime: Long,
  appSparkVersion: String = org.apache.spark.SPARK_VERSION): SparkUI

create(

sc: Option[SparkContext],

store: AppStatusStore,

conf: SparkConf,

securityManager: SecurityManager,

appName: String,

basePath: String = "",

startTime: Long,

appSparkVersion: String = org.apache.spark.SPARK_VERSION): SparkUI

create creates a SparkUI backed by a AppStatusStore.

Internally, create simply creates a new SparkUI (with the predefined Spark version).

Note	`create` is used when: `SparkContext` is created (for a running Spark application) `FsHistoryProvider` is requested to getAppUI (for a Spark application that already finished)

Creating SparkUI Instance

SparkUI takes the following when created:

AppStatusStore
SparkContext
SparkConf
SecurityManager
Application name
basePath
Start time
appSparkVersion

SparkUI initializes the internal registries and counters and the tabs and handlers.

Attaching Tabs and Context Handlers — `initialize` Method



initialize(): Unit

initialize(): Unit

Note	`initialize` is part of WebUI Contract to initialize web components.

initialize creates and attaches the following tabs (with the reference to the SparkUI and its AppStatusStore):

In the end, initialize creates and attaches the following ServletContextHandlers:

Creates a static handler for serving files from a static directory, i.e. /static to serve static files from org/apache/spark/ui/static directory (on CLASSPATH)
Creates a redirect handler to redirect / to /jobs/ (and so the Jobs tab is the welcome tab when you open the web UI)
Creates the /api/* context handler for the Status REST API
Creates a redirect handler to redirect /jobs/job/kill to /jobs/ and request the JobsTab to execute handleKillRequest before redirection
Creates a redirect handler to redirect /stages/stage/kill to /stages/ and request the StagesTab to execute handleKillRequest before redirection

SparkUI — Web UI of Spark Application