spark技术分享OrcFileFormat
OrcFileFormat is a FileFormat that…FIXME
buildReaderWithPartitionValues Method
|
1 2 3 4 5 6 7 8 9 10 11 12 |
buildReaderWithPartitionValues( sparkSession: SparkSession, dataSchema: StructType, partitionSchema: StructType, requiredSchema: StructType, filters: Seq[Filter], options: Map[String, String], hadoopConf: Configuration): (PartitionedFile) => Iterator[InternalRow] |
|
Note
|
buildReaderWithPartitionValues is part of FileFormat Contract to build a data reader with partition column values appended.
|
buildReaderWithPartitionValues…FIXME
inferSchema Method
|
1 2 3 4 5 6 7 8 |
inferSchema( sparkSession: SparkSession, options: Map[String, String], files: Seq[FileStatus]): Option[StructType] |
|
Note
|
inferSchema is part of FileFormat Contract to…FIXME.
|
inferSchema…FIXME
Building Partitioned Data Reader — buildReader Method
|
1 2 3 4 5 6 7 8 9 10 11 12 |
buildReader( sparkSession: SparkSession, dataSchema: StructType, partitionSchema: StructType, requiredSchema: StructType, filters: Seq[Filter], options: Map[String, String], hadoopConf: Configuration): (PartitionedFile) => Iterator[InternalRow] |
|
Note
|
buildReader is part of FileFormat Contract to…FIXME
|
buildReader…FIXME
