OffsetSeqMetadata-spark技术分享

OffsetSeqMetadata

OffsetSeqMetadata holds the metadata for…FIXME:

batchWatermarkMs – the current event time watermark in milliseconds (used to bound the lateness of data that will processed)
batchTimestampMs – the current batch processing timestamp in milliseconds
conf – batch configuration with spark.sql.shuffle.partitions and spark.sql.streaming.stateStore.providerClass Spark properties

Note	`OffsetSeqMetadata` is used mainly when `IncrementalExecution` is created.

OffsetSeqMetadata defines the following relevantSQLConfs:

relevantSQLConfs are used when OffsetSeqMetadata is created and is requested to setSessionConf.



apply(
  batchWatermarkMs: Long,
  batchTimestampMs: Long,
  sessionConf: RuntimeConfig): OffsetSeqMetadata

apply(

batchWatermarkMs: Long,

batchTimestampMs: Long,

sessionConf: RuntimeConfig): OffsetSeqMetadata

apply…FIXME

Note	`apply` is used when…FIXME



setSessionConf(metadata: OffsetSeqMetadata, sessionConf: RuntimeConfig): Unit

setSessionConf(metadata: OffsetSeqMetadata, sessionConf: RuntimeConfig): Unit

setSessionConf…FIXME

Note	`setSessionConf` is used when…FIXME