AppStatusListener-spark技术分享

AppStatusListener

AppStatusListener is a SparkListener that AppStatusStore uses to…FIXME

AppStatusListener is created when:

AppStatusStore is requested to createLiveStore (with the live flag enabled)
FsHistoryProvider is requested to rebuildAppStore (with the live flag disabled)

Table 1. AppStatusListener’s Event Handlers
Event	Handler
`SparkListenerApplicationStart`	onApplicationStart
`SparkListenerApplicationEnd`	onApplicationEnd
`SparkListenerBlockManagerAdded`	onBlockManagerAdded
`SparkListenerBlockManagerRemoved`	onBlockManagerRemoved
`SparkListenerBlockUpdated`	onBlockUpdated
`SparkListenerEnvironmentUpdate`	onEnvironmentUpdate
`SparkListenerEvent`	onOtherEvent
`SparkListenerExecutorAdded`	onExecutorAdded
`SparkListenerExecutorBlacklisted`	onExecutorBlacklisted
`SparkListenerExecutorMetricsUpdate`	onExecutorMetricsUpdate
`SparkListenerExecutorRemoved`	onExecutorRemoved
`SparkListenerExecutorUnblacklisted`	onExecutorUnblacklisted
`SparkListenerJobStart`	onJobStart
`SparkListenerJobEnd`	onJobEnd
`SparkListenerNodeBlacklisted`	onNodeBlacklisted
`SparkListenerNodeUnblacklisted`	onNodeUnblacklisted
`SparkListenerStageCompleted`	onStageCompleted
`SparkListenerStageSubmitted`	onStageSubmitted
`SparkListenerTaskEnd`	onTaskEnd
`SparkListenerTaskGettingResult`	onTaskGettingResult
`SparkListenerTaskStart`	onTaskStart
`SparkListenerUnpersistRDD`	onUnpersistRDD

Table 2. AppStatusListener’s Internal Properties (e.g. Registries, Counters and Flags)
Name	Description
`appInfo`	`v1.ApplicationInfo`
`appSummary`	`AppSummary`
`liveUpdatePeriodNs`
`coresPerTask`	Default: `1`
`liveRDDs`	LiveRDDs by RDD ID
`liveStages`	`LiveStages` by `(Int, Int)`
`liveTasks`	`LiveTask` by task ID
`liveJobs`	`LiveJob` by job ID
`liveExecutors`	`LiveExecutor` by executor ID
`pools`	`SchedulerPool` by FIXME
`activeExecutorCount`	Number of active executors

`onStageSubmitted` Method



onStageSubmitted(event: SparkListenerStageSubmitted): Unit

onStageSubmitted(event: SparkListenerStageSubmitted): Unit

Note	`onStageSubmitted` is part of SparkListener Contract to…FIXME.

onStageSubmitted…FIXME

`update` Internal Method



update(entity: LiveEntity, now: Long, last: Boolean = false): Unit

update(entity: LiveEntity, now: Long, last: Boolean = false): Unit

update simply requests the LiveEntity to write (with the ElementTrackingStore as the store and the last flag as checkTriggers flag).

Note	`update` is used in event handlers (i.e. `onApplicationStart`, `onExecutorRemoved`, `onJobEnd`, `onStageSubmitted`, `onTaskEnd`, `onStageCompleted`), liveUpdate, maybeUpdate, flush and updateRDDBlock.

`flush` Internal Method



flush(): Unit

flush(): Unit

flush…FIXME

Note	`flush` is used when…FIXME

`maybeUpdate` Internal Method



maybeUpdate(entity: LiveEntity, now: Long): Unit

maybeUpdate(entity: LiveEntity, now: Long): Unit

maybeUpdate…FIXME

Note	`maybeUpdate` is used when…FIXME

`liveUpdate` Internal Method



liveUpdate(entity: LiveEntity, now: Long): Unit

liveUpdate(entity: LiveEntity, now: Long): Unit

liveUpdate…FIXME

Note	`liveUpdate` is used when…FIXME

`updateStreamBlock` Internal Method



updateStreamBlock(event: SparkListenerBlockUpdated, stream: StreamBlockId): Unit

updateStreamBlock(event: SparkListenerBlockUpdated, stream: StreamBlockId): Unit

updateStreamBlock…FIXME

Note	`updateStreamBlock` is used exclusively when `AppStatusListener` is requested to handle a SparkListenerBlockUpdated event (for a StreamBlockId).

Intercepting SparkListenerBlockUpdated Events — `onBlockUpdated` Handler Method



onBlockUpdated(event: SparkListenerBlockUpdated): Unit

onBlockUpdated(event: SparkListenerBlockUpdated): Unit

Note	`onBlockUpdated` is part of SparkListener Contract to…FIXME.

onBlockUpdated simply dispatches to the following event-specific handlers (per BlockId type):

updateRDDBlock for RDDBlockIds
updateStreamBlock for StreamBlockIds
Ignores (swallows) the SparkListenerBlockUpdated event for the other types

Creating AppStatusListener Instance

AppStatusListener takes the following when created:

ElementTrackingStore
SparkConf
live flag
Optional lastUpdateTime (default: None)

AppStatusListener initializes the internal registries and counters.

`updateRDDBlock` Internal Method



updateRDDBlock(event: SparkListenerBlockUpdated, block: RDDBlockId): Unit

updateRDDBlock(event: SparkListenerBlockUpdated, block: RDDBlockId): Unit

updateRDDBlock…FIXME

Note	`updateRDDBlock` is used exclusively when `AppStatusListener` is requested to handle a SparkListenerBlockUpdated event (for a RDDBlockId).

AppStatusListener