spark技术分享StreamingDeduplicationStrategy Execution Planning Strategy for Deduplicate Logical Operator
StreamingDeduplicationStrategy is an execution planning strategy (i.e. Strategy) that IncrementalExecution uses to plan Deduplicate logical operators in streaming Datasets.
|
Note
|
Deduplicate logical operator is the result of dropDuplicates operator. |
StreamingDeduplicationStrategy is available using SessionState.
|
1 2 3 4 5 |
spark.sessionState.planner.StreamingDeduplicationStrategy |
StreamingDeduplicationStrategy resolves streaming Deduplicate unary logical operators to StreamingDeduplicateExec physical operators.
|
1 2 3 4 5 |
FIXME |
