ReplayListenerBus-spark技术分享

ReplayListenerBus

Note	`ReplayListenerBus` is used in FsHistoryProvider.

Note	`ReplayListenerBus` is a `private[spark]` class in `org.apache.spark.scheduler` package.



replay(
  logData: InputStream,
  sourceName: String,
  maybeTruncated: Boolean = false): Unit

replay(

logData: InputStream,

sourceName: String,

maybeTruncated: Boolean = false): Unit

replay reads JSON-encoded SparkListenerEvent events from logData (one event per line) and posts them to all registered SparkListenerInterface listeners.

Note	`replay` uses jackson from json4s library to parse the AST for JSON.

When there is an exception parsing a JSON event, you may see the following WARN message in the logs (for the last line) or a JsonParseException.



WARN Got JsonParseException from log file $sourceName at line [lineNumber], the file might not have finished writing cleanly.

WARN Got JsonParseException from log file $sourceName at line [lineNumber], the file might not have finished writing cleanly.

Any other non-IO exceptions end up with the following ERROR messages in the logs:



ERROR Exception parsing Spark event log: [sourceName]
ERROR Malformed line #[lineNumber]: [currentLine]

ERROR Exception parsing Spark event log: [sourceName]

ERROR Malformed line #[lineNumber]: [currentLine]

Note	The `sourceName` input argument is only used for messages.