StreamingQueryListener — Intercepting Streaming Events-spark技术分享

StreamingQueryListener — Intercepting Streaming Events

StreamingQueryListener is the contract for listeners that want to be notified about the life cycle events of streaming queries, i.e. start, progress and termination of a query.



package org.apache.spark.sql.streaming

abstract class StreamingQueryListener {
  def onQueryStarted(event: QueryStartedEvent): Unit
  def onQueryProgress(event: QueryProgressEvent): Unit
  def onQueryTerminated(event: QueryTerminatedEvent): Unit
}

package org.apache.spark.sql.streaming

abstract class StreamingQueryListener {

def onQueryStarted(event: QueryStartedEvent): Unit

def onQueryProgress(event: QueryProgressEvent): Unit

def onQueryTerminated(event: QueryTerminatedEvent): Unit

}

Table 1. StreamingQueryListener’s Life Cycle Events and Callbacks
Event	Callback	When Posted
`QueryStartedEvent` `id` `runId` `name`	`onQueryStarted`	Right after `StreamExecution` has started running streaming batches.
`QueryProgressEvent` StreamingQueryProgress	`onQueryProgress`	`ProgressReporter` reports query progress (which is when `StreamExecution` runs batches and a trigger has finished).
`QueryTerminatedEvent` `id` `runId` Optional `exception` if terminated due to an error	`onQueryTerminated`	Right before `StreamExecution` finishes running streaming batches (due to a stop or an exception).

You can register a StreamingQueryListener using StreamingQueryManager.addListener method.



val queryListener: StreamingQueryListener = ...
spark.streams.addListener(queryListener)

val queryListener: StreamingQueryListener = ...

spark.streams.addListener(queryListener)

You can remove a StreamingQueryListener using StreamingQueryManager.removeListener method.



val queryListener: StreamingQueryListener = ...
spark.streams.removeListener(queryListener)

val queryListener: StreamingQueryListener = ...

spark.streams.removeListener(queryListener)

StreamingQueryListener onQueryStarted.png

Figure 1. StreamingQueryListener Notified about Query’s Start (onQueryStarted)

Note	`onQueryStarted` is used internally to unblock the starting thread of `StreamExecution`.

StreamingQueryListener onQueryProgress.png

Figure 2. StreamingQueryListener Notified about Query’s Progress (onQueryProgress)

StreamingQueryListener onQueryTerminated.png

Figure 3. StreamingQueryListener Notified about Query’s Termination (onQueryTerminated)

Note	You can also register a streaming event listener using the general `SparkListener` interface. Details on `SparkListener` interface can be found in the Mastering Apache Spark 2 gitbook.

StreamingQueryListener — Intercepting Streaming Events

StreamingQueryListener — Intercepting Streaming Events

相关推荐

欢迎关注：spark技术分享

热门标签

近期文章

分类目录

关注公众号：spark技术分享

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏

QQ咨询

回顶部