StateStoreWriter Contract — Stateful Physical Operators That Write to StateStore-spark技术分享

StateStoreWriter Contract — Stateful Physical Operators That Write to StateStore

StateStoreWriter is the extension of the StatefulOperator Contract for physical operators that write to a state store and collect the write metrics for execution progress reporting.

Table 1. StateStoreWriter’s Performance Metrics
Key	Name (in web UI)	Description
`numOutputRows`	number of output rows
`numTotalStateRows`	number of total state rows
`numUpdatedStateRows`	number of updated state rows
`allUpdatesTimeMs`	total time to update rows
`allRemovalsTimeMs`	total time to remove rows
`commitTimeMs`	time to commit changes
`stateMemory`	memory used by state

Table 2. StateStoreWriters
StateStoreWriter	Description
FlatMapGroupsWithStateExec
StateStoreSaveExec
StreamingDeduplicateExec
StreamingGlobalLimitExec
StreamingSymmetricHashJoinExec

Setting StateStore-Specific Metrics for Stateful Physical Operator — `setStoreMetrics` Method



setStoreMetrics(store: StateStore): Unit

setStoreMetrics(store: StateStore): Unit

setStoreMetrics requests store for metrics to use them to record the following metrics of a physical operator:

numTotalStateRows as StateStore.numKeys
stateMemory as StateStore.memoryUsedBytes

setStoreMetrics records the implementation-specific metrics.

Note	`setStoreMetrics` is used when the following physical operators are executed: FlatMapGroupsWithStateExec StateStoreSaveExec StreamingDeduplicateExec StreamingGlobalLimitExec

`getProgress` Method



getProgress(): StateOperatorProgress

getProgress(): StateOperatorProgress

getProgress…FIXME

Note	`getProgress` is used exclusively when `ProgressReporter` is requested to extractStateOperatorMetrics (when `MicroBatchExecution` is requested to run the activated streaming query).

`shouldRunAnotherBatch` Method



shouldRunAnotherBatch(newMetadata: OffsetSeqMetadata): Boolean

shouldRunAnotherBatch(newMetadata: OffsetSeqMetadata): Boolean

shouldRunAnotherBatch is disabled by default.

Note	`shouldRunAnotherBatch` is used exclusively when `IncrementalExecution` is requested to shouldRunAnotherBatch (when `MicroBatchExecution` is requested to run the activated streaming query).

`stateStoreCustomMetrics` Internal Method



stateStoreCustomMetrics: Map[String, SQLMetric]

stateStoreCustomMetrics: Map[String, SQLMetric]

stateStoreCustomMetrics…FIXME

Note	`stateStoreCustomMetrics` is used when `StateStoreWriter` is requested for the metrics and getProgress.

`timeTakenMs` Method



timeTakenMs(body: => Unit): Long

timeTakenMs(body: => Unit): Long

timeTakenMs…FIXME

Note	`timeTakenMs` is used when…FIXME

StateStoreWriter Contract — Stateful Physical Operators That Write to StateStore