StreamingQueryProgress-spark技术分享

StreamingQueryProgress

StreamingQueryProgress holds information about the progress of a streaming query.

StreamingQueryProgress is created exclusively when StreamExecution finishes a trigger.

Note

Use lastProgress property of a StreamingQuery to access the most recent StreamingQueryProgress update.



val sq: StreamingQuery = ...
sq.lastProgress

val sq: StreamingQuery = ...

sq.lastProgress

Note

Use recentProgress property of a StreamingQuery to access the most recent StreamingQueryProgress updates.



val sq: StreamingQuery = ...
sq.recentProgress

val sq: StreamingQuery = ...

sq.recentProgress

Note	Use StreamingQueryListener to get notified about `StreamingQueryProgress` updates while a streaming query is executed.

Table 1. StreamingQueryProgress’s Properties
Name	Description
`id`	Unique identifier of a streaming query
`runId`	Unique identifier of the current execution of a streaming query
`name`	Optional query name
`timestamp`	Time when the trigger has started (in ISO8601 format).
`batchId`	Unique id of the current batch
`durationMs`	Durations of the internal phases (in milliseconds)
`eventTime`	Statistics of event time seen in this batch
`stateOperators`	Information about stateful operators in the query that store state.
`sources`	Statistics about the data read from every streaming source in a streaming query
`sink`	Information about progress made for a sink