spark技术分享TaskMetrics
TaskMetrics is a collection of metrics tracked during execution of a Task.
TaskMetrics uses accumulators to represent the metrics and offers “increment” methods to increment them.
|
Note
|
The local values of the accumulators for a task (as accumulated while the task runs) are sent from the executor to the driver when the task completes (and DAGScheduler re-creates TaskMetrics).
|
| Property | Name | Type | Description |
|---|---|---|---|
|
|
|
Used in memoryBytesSpilled, incMemoryBytesSpilled |
|
|
|
|
Used in updatedBlockStatuses, recording updated |
| Name | Description |
|---|---|
|
Internal accumulators indexed by their names. Used when NOTE: |
|
|
Collection of internal AccumulatorV2 objects. Used when…FIXME NOTE: |
|
|
Collection of external AccumulatorV2 objects. Used when NOTE: |
accumulators Method
|
Caution
|
FIXME |
mergeShuffleReadMetrics Method
|
Caution
|
FIXME |
memoryBytesSpilled Method
|
Caution
|
FIXME |
updatedBlockStatuses Method
|
Caution
|
FIXME |
setExecutorCpuTime Method
|
Caution
|
FIXME |
setResultSerializationTime Method
|
Caution
|
FIXME |
setJvmGCTime Method
|
Caution
|
FIXME |
setExecutorRunTime Method
|
Caution
|
FIXME |
setExecutorDeserializeCpuTime Method
|
Caution
|
FIXME |
setExecutorDeserializeTime Method
|
Caution
|
FIXME |
setUpdatedBlockStatuses Method
|
Caution
|
FIXME |
Re-Creating TaskMetrics From AccumulatorV2s — fromAccumulators Method
|
1 2 3 4 5 |
fromAccumulators(accums: Seq[AccumulatorV2[_, _]]): TaskMetrics |
fromAccumulators creates a new TaskMetrics and registers accums as internal and external task metrics (using nameToAccums internal registry).
Internally, fromAccumulators creates a new TaskMetrics. It then splits accums into internal and external task metrics collections (using nameToAccums internal registry).
For every internal task metrics, fromAccumulators finds the metrics in nameToAccums internal registry (of the new TaskMetrics instance), copies metadata, and merges state.
In the end, fromAccumulators adds the external accumulators to the new TaskMetrics instance.
|
Note
|
fromAccumulators is used exclusively when DAGScheduler gets notified that a task has finished (and re-creates TaskMetrics).
|
Recording Memory Bytes Spilled — incMemoryBytesSpilled Method
|
1 2 3 4 5 |
incMemoryBytesSpilled(v: Long): Unit |
incMemoryBytesSpilled adds v to _memoryBytesSpilled task metrics.
|
Note
|
|
Recording Updated BlockStatus For Block — incUpdatedBlockStatuses Method
|
1 2 3 4 5 |
incUpdatedBlockStatuses(v: (BlockId, BlockStatus)): Unit |
incUpdatedBlockStatuses adds v in _updatedBlockStatuses internal registry.
|
Note
|
incUpdatedBlockStatuses is used exclusively when BlockManager does addUpdatedBlockStatusToTaskMetrics.
|
Registering Internal Accumulators — register Method
|
1 2 3 4 5 |
register(sc: SparkContext): Unit |
register registers the internal accumulators (from nameToAccums internal registry) with countFailedValues enabled (true).
|
Note
|
register is used exclusively when Stage is requested for its new attempt.
|
