SQLConf — Internal Configuration Store-spark技术分享

SQLConf — Internal Configuration Store

SQLConf is an internal key-value configuration store for parameters and hints used in Spark Structured Streaming (and Spark SQL in general).

Table 1. SQLConf’s Property Methods
Name	Description
disabledV2StreamingMicroBatchReaders spark.sql.streaming.disabledV2MicroBatchReaders	Used exclusively when `MicroBatchExecution` is requested for analyzed logical plan
`FLATMAPGROUPSWITHSTATE_STATE_FORMAT_VERSION` spark.sql.streaming.flatMapGroupsWithState.stateFormatVersion
`SHUFFLE_PARTITIONS` `spark.sql.shuffle.partitions`	See spark.sql.shuffle.partitions in The Internals of Spark SQL.
`stateStoreProviderClass` spark.sql.streaming.stateStore.providerClass	Used when: `StateStoreWriter` is requested to stateStoreCustomMetrics (when `StateStoreWriter` is requested for the metrics and getProgress) `StateStoreConf` is created
`STREAMING_AGGREGATION_STATE_FORMAT_VERSION` spark.sql.streaming.aggregation.stateFormatVersion	Used when: StatefulAggregationStrategy execution planning strategy is executed `OffsetSeqMetadata` is requested for `relevantSQLConfs` and `relevantSQLConfDefaultValues`
`STREAMING_MULTIPLE_WATERMARK_POLICY` spark.sql.streaming.multipleWatermarkPolicy