BasicOperators-spark技术分享

BasicOperators Execution Planning Strategy

Table 1. BasicOperators’ Logical to Physical Operator Conversions
Logical Operator	Physical Operator
RunnableCommand	ExecutedCommandExec
MemoryPlan	LocalTableScanExec
DeserializeToObject	`DeserializeToObjectExec`
`SerializeFromObject`	`SerializeFromObjectExec`
`MapPartitions`	`MapPartitionsExec`
`MapElements`	`MapElementsExec`
`AppendColumns`	`AppendColumnsExec`
`AppendColumnsWithObject`	`AppendColumnsWithObjectExec`
`MapGroups`	`MapGroupsExec`
`CoGroup`	`CoGroupExec`
`Repartition` (with shuffle enabled)	ShuffleExchangeExec
`Repartition`	CoalesceExec
`SortPartitions`	SortExec
Sort	SortExec
`Project`	`ProjectExec`
`Filter`	FilterExec
`TypedFilter`	FilterExec
Expand	`ExpandExec`
Window	WindowExec
`Sample`	`SampleExec`
LocalRelation	LocalTableScanExec
`LocalLimit`	`LocalLimitExec`
`GlobalLimit`	`GlobalLimitExec`
`Union`	`UnionExec`
Generate	GenerateExec
`OneRowRelation`	RDDScanExec
`Range`	RangeExec
`RepartitionByExpression`	ShuffleExchangeExec
ExternalRDD	ExternalRDDScanExec
LogicalRDD	RDDScanExec

Tip	Confirm the operator mapping in the source code of `BasicOperators`.

Note	`BasicOperators` expects that `Distinct`, `Intersect`, and `Except` logical operators are not used in a logical plan and throws a `IllegalStateException` if not.