RDDStorageInfo contains information about RDD persistence:
-
RDD id
-
RDD name
-
Number of RDD partitions
-
Number of cached RDD partitions
-
Storage level ID
-
Memory used
-
Disk used
-
Data distribution (as Seq[RDDDataDistribution])
-
Partitions (as Seq[RDDPartitionInfo]])
RDDStorageInfo is created exclusively when LiveRDD is requested to doUpdate (when requested to write).
RDDStorageInfo is used when:
-
web UI’s StoragePage is requested to render an HTML table row and an entire table for RDD details
-
REST API’s AbstractApplicationResource is requested for rddList (at storage/rdd path)
-
AppStatusStore is requested for rddList
spark技术分享
