spark技术分享BroadcastFactory — Pluggable Broadcast Variable Factories
BroadcastFactory is the contract for factories of broadcast variables in Apache Spark.
|
Note
|
As of Spark 2.0, it is no longer possible to plug a custom BroadcastFactory in, and TorrentBroadcastFactory is the one and only known implementation.
|
BroadcastFactory is exclusively used and instantiated inside of BroadcastManager.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
package org.apache.spark.broadcast trait BroadcastFactory { def initialize(isDriver: Boolean, conf: SparkConf, securityMgr: SecurityManager): Unit def newBroadcast[T: ClassTag](value: T, isLocal: Boolean, id: Long): Broadcast[T] def unbroadcast(id: Long, removeFromDriver: Boolean, blocking: Boolean): Unit def stop(): Unit } |
|
Note
|
BroadcastFactory is a private[spark] contract.
|
| Method | Description |
|---|---|
|
|
Used when…FIXME |
|
|
Used when…FIXME |
|
|
Used when…FIXME |
|
|
Used when…FIXME |
|
Note
|
TorrentBroadcastFactory is the one and only known implementation of BroadcastFactory contract in Apache Spark. |
