AvroOptions — Avro Data Source Options-spark技术分享

AvroOptions — Avro Data Source Options

AvroOptions represents the options of the Avro data source.

Option / Key Default Value Description

avroSchema

(undefined)

Avro schema in JSON format

compression

(undefined)

Specifies the compression codec to use when writing Avro data to disk

Note	If the option is not defined explicitly, Avro data source uses spark.sql.avro.compression.codec configuration property.

ignoreExtension

false

Controls whether Avro data source should read all Avro files regardless of their extension (true) or not (false)

By default, Avro data source reads only files with .avro file extension.

Note	If the option is not defined explicitly, Avro data source uses `avro.mapred.ignore.inputs.without.extension` Hadoop runtime property.

recordName

topLevelRecord

Top-level record name when writing Avro data to disk

recordNamespace

(empty)

Record namespace when writing Avro data to disk

Note	The options are case-insensitive.

AvroOptions is created when AvroFileFormat is requested to inferSchema, prepareWrite and buildReader.

AvroOptions takes the following when created: