spark技术分享Map/Reduce-side Aggregator
Aggregator is a set of functions used to aggregate distributed data sets:
|
1 2 3 4 5 6 7 |
createCombiner: V => C mergeValue: (C, V) => C mergeCombiners: (C, C) => C |
|
Note
|
Aggregator is created in combineByKeyWithClassTag transformations to create ShuffledRDDs and is eventually passed on to ShuffleDependency. It is also used in ExternalSorter.
|
updateMetrics Internal Method
|
Caution
|
FIXME |
combineValuesByKey Method
|
Caution
|
FIXME |
combineCombinersByKey Method
|
Caution
|
FIXME |
