spark技术分享ShuffleMetricsSource — Metrics Source of BlockManager for Shuffle-Related Metrics
ShuffleMetricsSource is the metrics source of a BlockManager for shuffle-related metrics.
ShuffleMetricsSource lives on a Spark executor and is registered only when a Spark application runs in a non-local / cluster mode.
ShuffleMetricsSource is created exclusively when BlockManager is requested for the shuffleMetricsSource.
When created, ShuffleMetricsSource gets a MetricSet that BlockManager requests from the ShuffleClient (only when in a non-local / cluster mode).
ShuffleMetricsSource is registered under the following source name per the type of a BlockManager:
-
NettyBlockTransfer when spark.shuffle.service.enabled configuration property is off (
false) -
ExternalShuffle when spark.shuffle.service.enabled configuration property is on (
true)
|
Note
|
spark.shuffle.service.enabled configuration property is off (false) by default.
|
| Name | Type | Description |
|---|
|
Note
|
Since Executor does not have a web UI attached you cannot access the metrics using the HTTP protocol (through MetricsServlet JSON metrics sink).
|
Creating ShuffleMetricsSource Instance
ShuffleMetricsSource takes the following when created:
-
Dropwizard Metrics’ MetricSet
