Distribution — Contract For Data Distribution Across Partitions-spark技术分享

Distribution — Contract For Data Distribution Across Partitions

Distribution is the contract of…FIXME



package org.apache.spark.sql.catalyst.plans.physical

sealed trait Distribution {
  def requiredNumPartitions: Option[Int]
  def createPartitioning(numPartitions: Int): Partitioning
}

package org.apache.spark.sql.catalyst.plans.physical

sealed trait Distribution {

def requiredNumPartitions: Option[Int]

def createPartitioning(numPartitions: Int): Partitioning

}

Note	`Distribution` is a Scala `sealed` contract which means that all possible distributions are all in the same compilation unit (file).

Method Description

requiredNumPartitions

Gives the required number of partitions for a distribution.

Used exclusively when EnsureRequirements physical optimization is requested to enforce partition requirements of a physical operator (and a child operator’s output partitioning does not satisfy a required child distribution that leads to inserting a ShuffleExchangeExec operator to a physical plan).

Note	`None` for the required number of partitions indicates to use any number of partitions (possibly spark.sql.shuffle.partitions configuration property with the default of `200` partitions).

createPartitioning

Creates a Partitioning for a given number of partitions.

Used exclusively when EnsureRequirements physical optimization is requested to enforce partition requirements of a physical operator (and creates a ShuffleExchangeExec physical operator with a required Partitioning).

Table 2. Distributions
Distribution	Description
AllTuples
BroadcastDistribution
ClusteredDistribution
HashClusteredDistribution
OrderedDistribution
UnspecifiedDistribution

Distribution — Contract For Data Distribution Across Partitions

Distribution — Contract For Data Distribution Across Partitions

相关推荐

欢迎关注：spark技术分享

热门标签

近期文章

分类目录

关注公众号：spark技术分享

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏

QQ咨询

回顶部