spark技术分享CompressionCodec
With spark.broadcast.compress enabled (which is the default), TorrentBroadcast uses compression for broadcast blocks.
|
Caution
|
FIXME What’s compressed? |
| Codec Alias | Fully-Qualified Class Name | Notes |
|---|---|---|
|
|
|
The default implementation |
|
|
|
|
|
|
|
The fallback when the default codec is not available. |
An implementation of CompressionCodec trait has to offer a constructor that accepts a single argument being SparkConf. Read Creating CompressionCodec — createCodec Factory Method in this document.
You can control the default compression codec in a Spark application using spark.io.compression.codec Spark property.
Creating CompressionCodec — createCodec Factory Method
|
1 2 3 4 5 6 |
createCodec(conf: SparkConf): CompressionCodec (1) createCodec(conf: SparkConf, codecName: String): CompressionCodec (2) |
createCodec uses the internal shortCompressionCodecNames lookup table to find the input codecName (regardless of the case).
createCodec finds the constructor of the compression codec’s implementation (that accepts a single argument being SparkConf).
If a compression codec could not be found, createCodec throws a IllegalArgumentException exception:
|
1 2 3 4 5 |
Codec [<codecName>] is not available. Consider setting spark.io.compression.codec=snappy |
getCodecName Method
|
1 2 3 4 5 |
getCodecName(conf: SparkConf): String |
getCodecName reads spark.io.compression.codec Spark property from the input conf SparkConf or assumes lz4.
|
Note
|
getCodecName is used when SparkContext sets up event logging (for History Server) or when creating a CompressionCodec.
|
Settings
| Name | Default value | Description |
|---|---|---|
|
|
The compression codec to use. Used when getCodecName is called to find the current compression codec. |
