Partitioning — Specification of Physical Operator’s Output Partitions-spark技术分享

Partitioning — Specification of Physical Operator’s Output Partitions

Partitioning is the contract to hint the Spark Physical Optimizer for the number of partitions the output of a physical operator should be split across.



numPartitions: Int

numPartitions: Int

numPartitions is used in:

EnsureRequirements physical preparation rule to enforce partition requirements of a physical operator
SortMergeJoinExec for outputPartitioning for FullOuter join type
Partitioning.allCompatible

Table 1. Partitioning Schemes (Partitionings) and Their Properties
Partitioning	compatibleWith	guarantees	numPartitions	satisfies
`BroadcastPartitioning`	`BroadcastPartitioning` with the same `BroadcastMode`	Exactly the same `BroadcastPartitioning`	1	BroadcastDistribution with the same `BroadcastMode`
`HashPartitioning` `clustering` expressions `numPartitions`	`HashPartitioning` (when their underlying expressions are semantically equal, i.e. deterministic and canonically equal)	`HashPartitioning` (when their underlying expressions are semantically equal, i.e. deterministic and canonically equal)	Input `numPartitions`	UnspecifiedDistribution ClusteredDistribution with all the hashing expressions included in `clustering` expressions
`PartitioningCollection` `partitionings`	Any `Partitioning` that is compatible with one of the input `partitionings`	Any `Partitioning` that is guaranteed by any of the input `partitionings`	Number of partitions of the first `Partitioning` in the input `partitionings`	Any `Distribution` that is satisfied by any of the input `partitionings`
`RangePartitioning` `ordering` collection of `SortOrder` `numPartitions`	`RangePartitioning` (when semantically equal, i.e. underlying expressions are deterministic and canonically equal)	`RangePartitioning` (when semantically equal, i.e. underlying expressions are deterministic and canonically equal)	Input `numPartitions`	UnspecifiedDistribution OrderedDistribution with `requiredOrdering` that matches the input `ordering` ClusteredDistribution with all the children of the input `ordering` semantically equal to one of the `clustering` expressions
`RoundRobinPartitioning` `numPartitions`	Always negative	Always negative	Input `numPartitions`	UnspecifiedDistribution
`SinglePartition`	Any `Partitioning` with exactly one partition	Any `Partitioning` with exactly one partition	1	Any `Distribution` except BroadcastDistribution
`UnknownPartitioning` `numPartitions`	Always negative	Always negative	Input `numPartitions`	UnspecifiedDistribution

Partitioning — Specification of Physical Operator’s Output Partitions

Partitioning — Specification of Physical Operator’s Output Partitions

相关推荐

欢迎关注：spark技术分享

热门标签

近期文章

分类目录

关注公众号：spark技术分享

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏

QQ咨询

回顶部