spark技术分享Vectorized Parquet Reader
Vectorized Parquet Reader (aka Vectorized Parquet Decoding) allows for reading datasets in parquet format in batches, i.e. rows are decoded in batches. That aims at improving memory locality and cache utilization.
Quoting SPARK-12854 Vectorize Parquet reader:
The parquet encodings are largely designed to decode faster in batches, column by column. This can speed up the decoding considerably.
VectorizedParquetRecordReader is responsible for vectorized decoding and is used only when spark.sql.parquet.enableVectorizedReader configuration property is enabled and the result schema uses AtomicType data types only.
spark.sql.parquet.enableVectorizedReader Configuration Property
spark.sql.parquet.enableVectorizedReader configuration property is on by default.
|
1 2 3 4 5 6 |
val isParquetVectorizedReaderEnabled = spark.conf.get("spark.sql.parquet.enableVectorizedReader").toBoolean assert(isParquetVectorizedReaderEnabled, "spark.sql.parquet.enableVectorizedReader should be enabled by default") |
