spark技术分享AppStatusStore
AppStatusStore is…FIXME
AppStatusStore is created when:
-
SparkContextis created (that triggers creating a live store (i.e. an AppStatusStore for an active Spark application)) -
FsHistoryProvideris requested to create a LoadedAppUI
When created for a live Spark application, AppStatusStore creates a AppStatusListener that is later used to get the active stages.
streamBlocksList Method
|
1 2 3 4 5 |
streamBlocksList(): Seq[StreamBlockData] |
streamBlocksList…FIXME
|
Note
|
streamBlocksList is used when…FIXME
|
activeStages Method
|
1 2 3 4 5 |
activeStages(): Seq[v1.StageData] |
activeStages…FIXME
|
Note
|
activeStages is used when…FIXME
|
Creating AppStatusStore Instance
AppStatusStore takes the following when created:
-
Optional AppStatusListener (default:
None)
Creating Live Store (AppStatusStore For Active Spark Application) — createLiveStore Factory Method
|
1 2 3 4 5 |
createLiveStore(conf: SparkConf): AppStatusStore |
createLiveStore creates a fully-initialized AppStatusStore.
Internally, createLiveStore creates a ElementTrackingStore (with a new InMemoryStore and the input SparkConf).
createLiveStore creates a AppStatusListener (with the ElementTrackingStore created, the input SparkConf and the live flag enabled).
In the end, createLiveStore creates an AppStatusStore (with the ElementTrackingStore and AppStatusListener just created).
|
Note
|
createLiveStore is used exclusively when SparkContext is created.
|
rddList Method
|
1 2 3 4 5 |
rddList(cachedOnly: Boolean = true): Seq[v1.RDDStorageInfo] |
In the end, rddList takes RDDStorageInfos with at least one partition cached (when cachedOnly flag is on) or all RDDStorageInfos (when cachedOnly flag is off).
|
Note
|
cachedOnly flag is on and therefore rddList gives RDDs cached only.
|
|
Note
|
|
