spark技术分享BlockRDD
BlockRDD is an RDD that is created when Spark Streaming’s ReceiverInputDStream is requested to compute and createBlockRDD.
Spark Streaming calls BlockRDD.removeBlocks() while clearing metadata.
|
Note
|
It appears that BlockRDD is used in Spark Streaming exclusively.
|
Computing Partition (in TaskContext) — compute Method
|
1 2 3 4 5 |
compute(split: Partition, context: TaskContext): Iterator[T] |
|
Note
|
compute is part of RDD Contract to compute a partition (in a TaskContext).
|
compute…FIXME
getPartitions Method
|
1 2 3 4 5 |
getPartitions: Array[Partition] |
|
Note
|
getPartitions is part of RDD Contract to…FIXME.
|
getPartitions…FIXME
getPreferredLocations Method
|
1 2 3 4 5 |
getPreferredLocations(split: Partition): Seq[String] |
|
Note
|
getPreferredLocations is part of RDD Contract to…FIXME.
|
getPreferredLocations…FIXME
Creating BlockRDD Instance
BlockRDD takes the following when created:
-
Collection of BlockIds
BlockRDD initializes the internal registries and counters.
