spark技术分享BlockRDD
BlockRDD is an RDD that is created when Spark Streaming’s ReceiverInputDStream is requested to compute and createBlockRDD.
Spark Streaming calls BlockRDD.removeBlocks() while clearing metadata.
|
Note
|
It appears that BlockRDD is used in Spark Streaming exclusively.
|
Computing Partition (in TaskContext) — compute Method
|
Note
|
compute is part of RDD Contract to compute a partition (in a TaskContext).
|
compute…FIXME
getPreferredLocations Method
|
Note
|
getPreferredLocations is part of RDD Contract to…FIXME.
|
getPreferredLocations…FIXME
Creating BlockRDD Instance
BlockRDD takes the following when created:
-
Collection of BlockIds
BlockRDD initializes the internal registries and counters.
