CompressionCodecs-spark技术分享

CompressionCodecs

CompressionCodecs is a utility object…FIXME

Table 1. Known Compression Codecs
Alias	Fully-Qualified Class Name
`none`
`uncompressed`
`bzip2`	`org.apache.hadoop.io.compress.BZip2Codec`
`deflate`	`org.apache.hadoop.io.compress.DeflateCodec`
`gzip`	`org.apache.hadoop.io.compress.GzipCodec`
`lz4`	`org.apache.hadoop.io.compress.Lz4Codec`
`snappy`	`org.apache.hadoop.io.compress.SnappyCodec`



setCodecConfiguration(conf: Configuration, codec: String): Unit

setCodecConfiguration(conf: Configuration, codec: String): Unit

setCodecConfiguration sets compression-related configurations to the Hadoop Configuration per the input codec.

Note	The input `codec` should be a fully-qualified class name, i.e. `org.apache.hadoop.io.compress.SnappyCodec`.

If the input codec is defined (i.e. not null), setCodecConfiguration sets the following configuration properties.

Table 2. Compression-Related Hadoop Configuration Properties (codec defined)
Name	Value
`mapreduce.output.fileoutputformat.compress`	`true`
`mapreduce.output.fileoutputformat.compress.type`	`BLOCK`
`mapreduce.output.fileoutputformat.compress.codec`	The input `codec` name
`mapreduce.map.output.compress`	`true`
`mapreduce.map.output.compress.codec`	The input `codec` name

If the input codec is not defined (i.e. null), setCodecConfiguration sets the following configuration properties.

Table 3. Compression-Related Hadoop Configuration Properties (codec not defined)
Name	Value
`mapreduce.output.fileoutputformat.compress`	`false`
`mapreduce.map.output.compress`	`false`

Note	`setCodecConfiguration` is used when CSVFileFormat, JsonFileFormat and TextFileFormat are requested to `prepareWrite`.