关注 spark技术分享,
撸spark源码 玩spark最佳实践

Deduplicate Unary Logical Operator

Deduplicate Unary Logical Operator

Deduplicate is a unary logical operator (i.e. LogicalPlan) that is created to represent dropDuplicates operator (that drops duplicate records for a given subset of columns).

Deduplicate has streaming flag enabled for streaming Datasets.

Caution
FIXME Example with duplicates across batches to show that Deduplicate keeps state and withWatermark operator should also be used to limit how much is stored (to not cause OOM)
Note

UnsupportedOperationChecker ensures that dropDuplicates operator is not used after aggregation on streaming Datasets.

The following code is not supported in Structured Streaming and results in an AnalysisException.

Note

Deduplicate logical operator is translated (aka planned) to:

  • StreamingDeduplicateExec physical operator in StreamingDeduplicationStrategy execution planning strategy for streaming Datasets (aka streaming plans)

  • Aggregate physical operator in ReplaceDeduplicateWithAggregate execution planning strategy for non-streaming/batch Datasets (aka batch plans)

The output schema of Deduplicate is exactly the child‘s output schema.

Creating Deduplicate Instance

Deduplicate takes the following when created:

  • Attributes for keys

  • Child logical operator (i.e. LogicalPlan)

  • Flag whether the logical operator is for streaming (enabled) or batch (disabled) mode

赞(0) 打赏
未经允许不得转载:spark技术分享 » Deduplicate Unary Logical Operator
分享到: 更多 (0)

关注公众号:spark技术分享

联系我们联系我们

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏