BroadcastManager-spark技术分享

BroadcastManager

Broadcast Manager (BroadcastManager) is a Spark service to manage broadcast variables in Spark. It is created for a Spark application when SparkContext is initialized and is a simple wrapper around BroadcastFactory.

BroadcastManager tracks the number of broadcast variables in a Spark application (using the internal field nextBroadcastId).

The idea is to transfer values used in transformations from a driver to executors in a most effective way so they are copied once and used many times by tasks (rather than being copied every time a task is launched).

When initialized, BroadcastManager creates an instance of TorrentBroadcastFactory.

`stop` Method

Caution

FIXME

Creating `BroadcastManager` Instance

Caution

FIXME

Initializing BroadcastManager — `initialize` Internal Method



initialize(): Unit

initialize(): Unit

initialize creates and initializes a TorrentBroadcastFactory.

Note	`initialize` is executed only once (when `BroadcastManager` is created) and controlled by the internal `initialized` flag.

`newBroadcast` Method



newBroadcast[T](value_ : T, isLocal: Boolean): Broadcast[T]

newBroadcast[T](value_ : T, isLocal: Boolean): Broadcast[T]

newBroadcast simply requests the current BroadcastFactory for a new broadcast variable.

Note	The `BroadcastFactory` is created when `BroadcastManager` is initialized.

Note	`newBroadcast` is executed for `SparkContext.broadcast` method and when `MapOutputTracker` serializes `MapStatuses`.

Settings

Table 1. Settings
Name	Default value	Description
`spark.broadcast.blockSize`	`4m`	The size of a block (in kB when unit not specified). Used when `TorrentBroadcast` stores brodcast blocks to `BlockManager`.
`spark.broadcast.compress`	`true`	The flag to enable compression. Refer to CompressionCodec. Used when `TorrentBroadcast` is created and later when it stores broadcast blocks to `BlockManager`. Also in SerializerManager.

BroadcastManager