HistoryServer — WebUI For Active And Completed Spark Applications-spark技术分享

HistoryServer — WebUI For Active And Completed Spark Applications

HistoryServer is the web UI for running (active) and completed Spark applications.

HistoryServer is created when…FIXME

HistoryServer uses the HttpServlet to handle requests to /* URI that FIXME.

Table 1. HistoryServer’s Internal Properties (e.g. Registries, Counters and Flags)
Name	Description
`appCache`	ApplicationCache Used when…FIXME
`loaderServlet`	Java Servlets’ HttpServlet Used exclusively when `HistoryServer` is requested to initialize (and attaches the servlet to the web UI to handle `/*` URI)

Tip

Enable INFO logging level for org.apache.spark.deploy.history.HistoryServer logger to see what happens inside.

Add the following line to conf/log4j.properties:



log4j.logger.org.apache.spark.deploy.history.HistoryServer=INFO

log4j.logger.org.apache.spark.deploy.history.HistoryServer=INFO

Refer to Logging.

`attachSparkUI` Method



attachSparkUI(
  appId: String,
  attemptId: Option[String],
  ui: SparkUI,
  completed: Boolean): Unit

attachSparkUI(

appId: String,

attemptId: Option[String],

ui: SparkUI,

completed: Boolean): Unit

Note	`attachSparkUI` is part of ApplicationCacheOperations Contract to…FIXME.

attachSparkUI…FIXME

Initializing HistoryServer — `initialize` Method



initialize(): Unit

initialize(): Unit

Note	`initialize` is part of WebUI Contract to initialize web components.

initialize…FIXME

Launching HistoryServer Standalone Application — `main` Method



main(argStrings: Array[String]): Unit

main(argStrings: Array[String]): Unit

main…FIXME

Creating HistoryServer Instance

HistoryServer takes the following when created:

SparkConf
ApplicationHistoryProvider
SecurityManager
Port number

HistoryServer initializes the internal registries and counters.

Requesting Spark Application UI — `getAppUI` Method



getAppUI(appId: String, attemptId: Option[String]): Option[LoadedAppUI]

getAppUI(appId: String, attemptId: Option[String]): Option[LoadedAppUI]

Note	`getAppUI` is part of ApplicationCacheOperations Contract to…FIXME.

getAppUI…FIXME

`withSparkUI` Method



withSparkUI[T](appId: String, attemptId: Option[String])(fn: SparkUI => T): T

withSparkUI[T](appId: String, attemptId: Option[String])(fn: SparkUI => T): T

Note	`withSparkUI` is part of UIRoot Contract to…FIXME.

withSparkUI…FIXME

`loadAppUi` Internal Method



loadAppUi(appId: String, attemptId: Option[String]): Boolean

loadAppUi(appId: String, attemptId: Option[String]): Boolean

loadAppUi…FIXME

Note	`loadAppUi` is used exclusively when `HistoryServer` is created.

`doGet` Method



doGet(req: HttpServletRequest, res: HttpServletResponse): Unit

doGet(req: HttpServletRequest, res: HttpServletResponse): Unit

Note	`doGet` is part of Java Servlet’s HttpServlet to handle HTTP GET requests.

doGet…FIXME

Note	`doGet` is used when…FIXME

HistoryServer — WebUI For Active And Completed Spark Applications