AggregateProcessor-spark技术分享

AggregateProcessor

AggregateProcessor is created and used exclusively when WindowExec physical operator is executed.

AggregateProcessor supports DeclarativeAggregate and ImperativeAggregate aggregate functions only (which happen to be AggregateFunction in AggregateExpression or AggregateWindowFunction).

Table 1. AggregateProcessor’s Properties
Name	Description
`buffer`	`SpecificInternalRow` with data types given bufferSchema

Note	`AggregateProcessor` is created using `AggregateProcessor` factory object (using apply method).

`initialize` Method



initialize(size: Int): Unit

initialize(size: Int): Unit

Caution

FIXME

Note	`initialize` is used when: `SlidingWindowFunctionFrame` writes out to the target row `UnboundedWindowFunctionFrame` is prepared `UnboundedPrecedingWindowFunctionFrame` is prepared `UnboundedFollowingWindowFunctionFrame` writes out to the target row

`evaluate` Method



evaluate(target: InternalRow): Unit

evaluate(target: InternalRow): Unit

Caution

FIXME

Note	`evaluate` is used when…FIXME

`apply` Factory Method



apply(
  functions: Array[Expression],
  ordinal: Int,
  inputAttributes: Seq[Attribute],
  newMutableProjection: (Seq[Expression], Seq[Attribute]) => MutableProjection): AggregateProcessor

apply(

functions: Array[Expression],

ordinal: Int,

inputAttributes: Seq[Attribute],

newMutableProjection: (Seq[Expression], Seq[Attribute]) => MutableProjection): AggregateProcessor

Note	`apply` is used exclusively when `WindowExec` is executed (and creates WindowFunctionFrame per `AGGREGATE` window aggregate functions, i.e. AggregateExpression or AggregateWindowFunction)

Executing update on ImperativeAggregates — `update` Method



update(input: InternalRow): Unit

update(input: InternalRow): Unit

update executes the update method on every input ImperativeAggregate sequentially (one by one).

Internally, update joins buffer with input internal binary row and converts the joined InternalRow using the MutableProjection function.

update then requests every ImperativeAggregate to update passing in the buffer and the input input rows.

Note	`MutableProjection` mutates the same underlying binary row object each time it is executed.

Note	`update` is used when `WindowFunctionFrame` prepares or writes.

Creating AggregateProcessor Instance

AggregateProcessor takes the following when created:

Schema of the buffer (as a collection of AttributeReferences)
Initial MutableProjection
Update MutableProjection
Evaluate MutableProjection
ImperativeAggregate expressions for aggregate functions
Flag whether to track partition size

AggregateProcessor