SparkListenerBus — Internal Contract for Spark Event Buses-spark技术分享

SparkListenerBus — Internal Contract for Spark Event Buses

SparkListenerBus is a private[spark] ListenerBus for SparkListenerInterface listeners that process SparkListenerEvent events.

SparkListenerBus comes with a custom doPostEvent method that simply relays SparkListenerEvent events to appropriate SparkListenerInterface methods.

Note	There are two custom `SparkListenerBus` listeners: LiveListenerBus and ReplayListenerBus.

Table 1. SparkListenerEvent to SparkListenerInterface’s Method “mapping”
SparkListenerEvent	SparkListenerInterface’s Method
`SparkListenerStageSubmitted`	`onStageSubmitted`
`SparkListenerStageCompleted`	`onStageCompleted`
`SparkListenerJobStart`	`onJobStart`
`SparkListenerJobEnd`	`onJobEnd`
`SparkListenerJobEnd`	`onJobEnd`
`SparkListenerTaskStart`	`onTaskStart`
`SparkListenerTaskGettingResult`	`onTaskGettingResult`
SparkListenerTaskEnd	`onTaskEnd`
`SparkListenerEnvironmentUpdate`	`onEnvironmentUpdate`
`SparkListenerBlockManagerAdded`	`onBlockManagerAdded`
`SparkListenerBlockManagerRemoved`	`onBlockManagerRemoved`
`SparkListenerUnpersistRDD`	`onUnpersistRDD`
`SparkListenerApplicationStart`	`onApplicationStart`
`SparkListenerApplicationEnd`	`onApplicationEnd`
`SparkListenerExecutorMetricsUpdate`	`onExecutorMetricsUpdate`
`SparkListenerExecutorAdded`	`onExecutorAdded`
`SparkListenerExecutorRemoved`	`onExecutorRemoved`
`SparkListenerBlockUpdated`	`onBlockUpdated`
`SparkListenerLogStart`	event ignored
other event types	`onOtherEvent`

`ListenerBus` Event Bus Contract



ListenerBus[L <: AnyRef, E]

ListenerBus[L <: AnyRef, E]

ListenerBus is an event bus that post events (of type E) to all registered listeners (of type L).

It manages listeners of type L, i.e. it can add to and remove listeners from an internal listeners collection.



addListener(listener: L): Unit
removeListener(listener: L): Unit

addListener(listener: L): Unit

removeListener(listener: L): Unit

It can post events of type E to all registered listeners (using postToAll method). It simply iterates over the internal listeners collection and executes the abstract doPostEvent method.



doPostEvent(listener: L, event: E): Unit

doPostEvent(listener: L, event: E): Unit

Note	`doPostEvent` is provided by more specialized `ListenerBus` event buses.

In case of exception while posting an event to a listener you should see the following ERROR message in the logs and the exception.



ERROR Listener [listener] threw an exception

ERROR Listener [listener] threw an exception

Note	There are three custom `ListenerBus` listeners: SparkListenerBus, StreamingQueryListenerBus, and StreamingListenerBus.

Tip

Enable ERROR logging level for org.apache.spark.util.ListenerBus logger to see what happens inside.

Add the following line to conf/log4j.properties:



log4j.logger.org.apache.spark.util.ListenerBus=ERROR

log4j.logger.org.apache.spark.util.ListenerBus=ERROR

Refer to Logging.

SparkListenerBus — Internal Contract for Spark Event Buses