OnHeapColumnVector-spark技术分享

OnHeapColumnVector

OnHeapColumnVector is a concrete WritableColumnVector that…FIXME

OnHeapColumnVector is created when:

OnHeapColumnVector is requested to allocate column vectors and reserveNewColumn
OrcColumnarBatchReader is requested to initBatch

Allocating Column Vectors — `allocateColumns` Static Method



OnHeapColumnVector[] allocateColumns(int capacity, StructType schema) (1)
OnHeapColumnVector[] allocateColumns(int capacity, StructField[] fields)

OnHeapColumnVector[] allocateColumns(int capacity, StructType schema) (1)

OnHeapColumnVector[] allocateColumns(int capacity, StructField[] fields)

Simply converts StructType to StructField[] and calls the other allocateColumns

allocateColumns creates an array of OnHeapColumnVector for every field (to hold capacity number of elements of the data type per field).

Note

allocateColumns is used when:

AggregateHashMap is created
InMemoryTableScanExec is requested to createAndDecompressColumn
VectorizedParquetRecordReader is requested to initBatch (with ON_HEAP memory mode)
OrcColumnarBatchReader is requested to initBatch (with ON_HEAP memory mode)
ColumnVectorUtils is requested to convert an iterator of rows into a single ColumnBatch (aka toBatch)

Creating OnHeapColumnVector Instance

OnHeapColumnVector takes the following when created:

Number of elements to hold in a vector (aka capacity)
Data type of the elements stored

When created, OnHeapColumnVector reserveInternal (for the given capacity) and reset.

`reserveInternal` Method



void reserveInternal(int newCapacity)

void reserveInternal(int newCapacity)

Note	`reserveInternal` is part of WritableColumnVector Contract to…FIXME.

reserveInternal…FIXME

`reserveNewColumn` Method



OnHeapColumnVector reserveNewColumn(int capacity, DataType type)

OnHeapColumnVector reserveNewColumn(int capacity, DataType type)

Note	`reserveNewColumn` is part of WritableColumnVector Contract to…FIXME.

reserveNewColumn…FIXME

OnHeapColumnVector