关注 spark技术分享,
撸spark源码 玩spark最佳实践

GroupStateTimeout

GroupStateTimeout

GroupStateTimeout represents the possible timeouts that you can use for the state-aware Dataset operations:

GroupStateTimeout is part of org.apache.spark.sql.streaming package.

Table 1. Types of GroupStateTimeouts (in alphabetical order)
GroupStateTimeout Description

ProcessingTimeTimeout

Timeout based on the processing time.

Note

FlatMapGroupsWithStateExec requires that batchTimestampMs is specified when ProcessingTimeTimeout is used.

batchTimestampMs is defined when IncrementalExecution is created (and so is state). IncrementalExecution is given OffsetSeqMetadata when StreamExecution runs a streaming batch.

Caution
FIXME Describe OffsetSeqMetadata and StreamExecution.offsetSeqMetadata

EventTimeTimeout

Timeout based on the event time

Used when…​FIXME

NoTimeout

No timeout

Used when…​FIXME

赞(0) 打赏
未经允许不得转载:spark技术分享 » GroupStateTimeout
分享到: 更多 (0)

关注公众号:spark技术分享

联系我们联系我们

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏