关注 spark技术分享,
撸spark源码 玩spark最佳实践

SerializerManager

SerializerManager

Caution
FIXME

When SparkEnv is created (either for the driver or executors), it instantiates SerializerManager that is then used to create a BlockManager.

The common idiom in Spark’s code is to access the current SerializerManager using SparkEnv.

Note
SerializerManager was introduced in SPARK-13926.

Creating SerializerManager Instance

Caution
FIXME

wrapStream Method

Caution
FIXME

dataDeserializeStream Method

Caution
FIXME

Automatic Selection of Best Serializer

Caution
FIXME

SerializerManager will automatically pick a Kryo serializer for ShuffledRDDs whose key, value, and/or combiner types are primitives, arrays of primitives, or strings.

Selecting “Best” Serializer — getSerializer Method

getSerializer selects the “best” Serializer given the input types for keys and values (in a RDD).

getSerializer returns KryoSerializer when the types of keys and values are compatible with Kryo or the default Serializer.

Note
The default Serializer is defined when SerializerManager is created.

Settings

Table 1. Spark Properties
Name Default value Description

spark.shuffle.compress

true

The flag to control whether to compress shuffle output when stored

spark.rdd.compress

false

The flag to control whether to compress RDD partitions when stored serialized.

spark.shuffle.spill.compress

true

The flag to control whether to compress shuffle output temporarily spilled to disk.

spark.block.failures.beforeLocationRefresh

5

spark.io.encryption.enabled

false

The flag to enable IO encryption

赞(0) 打赏
未经允许不得转载:spark技术分享 » SerializerManager
分享到: 更多 (0)

关注公众号:spark技术分享

联系我们联系我们

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏