AggregationIterator — Generic Iterator of UnsafeRows for Aggregate Physical Operators-spark技术分享

AggregationIterator — Generic Iterator of UnsafeRows for Aggregate Physical Operators

AggregationIterator is the base for iterators of UnsafeRows that…FIXME

Iterators are data structures that allow to iterate over a sequence of elements. They have a hasNext method for checking if there is a next element available, and a next method which returns the next element and discards it from the iterator.

Name Description

ObjectAggregationIterator

Used exclusively when ObjectHashAggregateExec physical operator is executed.

SortBasedAggregationIterator

Used exclusively when SortAggregateExec physical operator is executed.

TungstenAggregationIterator

Used exclusively when HashAggregateExec physical operator is executed.

Note	HashAggregateExec operator is the preferred aggregate physical operator for Aggregation execution planning strategy (over `ObjectHashAggregateExec` and `SortAggregateExec`).

Table 2. AggregationIterator’s Internal Registries and Counters
Name	Description
`aggregateFunctions`	Aggregate functions Used when…FIXME
`allImperativeAggregateFunctions`	ImperativeAggregate functions Used when…FIXME
`allImperativeAggregateFunctionPositions`	Positions Used when…FIXME
`expressionAggInitialProjection`	`MutableProjection` Used when…FIXME
`generateOutput`	Function used to generate an unsafe row (i.e. `(UnsafeRow, InternalRow) ⇒ UnsafeRow`) Used when: `ObjectAggregationIterator` is requested for the next unsafe row and outputForEmptyGroupingKeyWithoutInput `SortBasedAggregationIterator` is requested for the next unsafe row and outputForEmptyGroupingKeyWithoutInput `TungstenAggregationIterator` is requested for the next unsafe row and outputForEmptyGroupingKeyWithoutInput
`groupingAttributes`	Grouping attributes Used when…FIXME
`groupingProjection`	UnsafeProjection Used when…FIXME
`processRow`	`(InternalRow, InternalRow) ⇒ Unit` Used when…FIXME

Creating AggregationIterator Instance

AggregationIterator takes the following when created:

Grouping named expressions
Input attributes
Aggregate expressions
Aggregate attributes
Initial input buffer offset
Result named expressions
Function to create a new MutableProjection given expressions and attributes

AggregationIterator initializes the internal registries and counters.

Note	`AggregationIterator` is a Scala abstract class and cannot be created directly. It is created indirectly for the concrete AggregationIterators.

`initializeAggregateFunctions` Internal Method



initializeAggregateFunctions(
  expressions: Seq[AggregateExpression],
  startingInputBufferOffset: Int): Array[AggregateFunction]

initializeAggregateFunctions(

expressions: Seq[AggregateExpression],

startingInputBufferOffset: Int): Array[AggregateFunction]

initializeAggregateFunctions…FIXME

Note	`initializeAggregateFunctions` is used when…FIXME

`generateProcessRow` Internal Method



generateProcessRow(
  expressions: Seq[AggregateExpression],
  functions: Seq[AggregateFunction],
  inputAttributes: Seq[Attribute]): (InternalRow, InternalRow) => Unit

generateProcessRow(

expressions: Seq[AggregateExpression],

functions: Seq[AggregateFunction],

inputAttributes: Seq[Attribute]): (InternalRow, InternalRow) => Unit

generateProcessRow…FIXME

Note	`generateProcessRow` is used when…FIXME

`generateResultProjection` Method



generateResultProjection(): (UnsafeRow, InternalRow) => UnsafeRow

generateResultProjection(): (UnsafeRow, InternalRow) => UnsafeRow

generateResultProjection…FIXME

Note	`generateResultProjection` is used when: `AggregationIterator` is created `TungstenAggregationIterator` is requested for the generateResultProjection

AggregationIterator — Generic Iterator of UnsafeRows for Aggregate Physical Operators