ShuffleWriteMetrics-spark技术分享

ShuffleWriteMetrics

ShuffleWriteMetrics is a collection of accumulators that represents task metrics about writing shuffle data.

ShuffleWriteMetrics tracks the following task metrics:

Shuffle Bytes Written
Shuffle Write Time
Shuffle Records Written

Note	Accumulators allow tasks (running on executors) to communicate with the driver.

Table 1. ShuffleWriteMetrics’s Accumulators
Name	Description
`_bytesWritten`	Accumulator to track how many shuffle bytes were written in a shuffle task. Used when `ShuffleWriteMetrics` is requested the shuffle bytes written and to increment or decrement it. NOTE: `_bytesWritten` is available as `internal.metrics.shuffle.write.bytesWritten` (internally `shuffleWrite.BYTES_WRITTEN`) in TaskMetrics.
`_writeTime`	Accumulator to track shuffle write time (as 64-bit integer) of a shuffle task. Used when `ShuffleWriteMetrics` is requested the shuffle write time and to increment it. NOTE: `_writeTime` is available as `internal.metrics.shuffle.write.writeTime` (internally `shuffleWrite.WRITE_TIME`) in TaskMetrics.
`_recordsWritten`	Accumulator to track how many shuffle records were written in a shuffle task. Used when `ShuffleWriteMetrics` is requested the shuffle records written and to increment or decrement it. NOTE: `_recordsWritten` is available as `internal.metrics.shuffle.write.recordsWritten` (internally `shuffleWrite.RECORDS_WRITTEN`) in TaskMetrics.

`decRecordsWritten` Method

Caution

FIXME

`decBytesWritten` Method

Caution

FIXME

`writeTime` Method

Caution

FIXME

`recordsWritten` Method

Caution

FIXME

Returning Number of Shuffle Bytes Written — `bytesWritten` Method



bytesWritten: Long

bytesWritten: Long

bytesWritten represents the shuffle bytes written metrics of a shuffle task.

Internally, bytesWritten returns the sum of _bytesWritten internal accumulator.

Note

bytesWritten is used when:

ShuffleWriteMetricsUIData is created
In decBytesWritten
StatsReportListener intercepts stage completed events to show shuffle bytes written
ShuffleExternalSorter does writeSortedFile (to incDiskBytesSpilled)
JsonProtocol converts ShuffleWriteMetrics to JSON
ExecutorsListener intercepts task end events to update executor metrics
JobProgressListener updates stage and executor metrics

Incrementing Shuffle Bytes Written Metrics — `incBytesWritten` Method



incBytesWritten(v: Long): Unit

incBytesWritten(v: Long): Unit

incBytesWritten simply adds v to _bytesWritten internal accumulator.

Note	`incBytesWritten` is used when: `UnsafeShuffleWriter` does `mergeSpills` `DiskBlockObjectWriter` does `updateBytesWritten` `JsonProtocol` creates `TaskMetrics` from JSON

Incrementing Shuffle Write Time Metrics — `incWriteTime` Method



incWriteTime(v: Long): Unit

incWriteTime(v: Long): Unit

incWriteTime simply adds v to _writeTime internal accumulator.

Note

incWriteTime is used when:

SortShuffleWriter stops.
BypassMergeSortShuffleWriter writes records (i.e. when it initializes DiskBlockObjectWriter partition writers) and later when concatenates per-partition files into a single file.
UnsafeShuffleWriter does mergeSpillsWithTransferTo.
DiskBlockObjectWriter does commitAndGet (but only when syncWrites flag is enabled that forces outstanding writes to disk).
JsonProtocol creates TaskMetrics from JSON
TimeTrackingOutputStream does its operation (after all it is an output stream to track shuffle write time).

Incrementing Shuffle Records Written Metrics — `incRecordsWritten` Method



incRecordsWritten(v: Long): Unit

incRecordsWritten(v: Long): Unit

incRecordsWritten simply adds v to _recordsWritten internal accumulator.

Note	`incRecordsWritten` is used when: `ShuffleExternalSorter` does `writeSortedFile` `DiskBlockObjectWriter` does `recordWritten` `JsonProtocol` creates `TaskMetrics` from JSON

ShuffleWriteMetrics