spark技术分享MetricsConfig — Metrics System Configuration
MetricsConfig is the configuration of the MetricsSystem (i.e. metrics sources and sinks).
MetricsConfig is created when MetricsSystem is.
MetricsConfig uses metrics.properties as the default metrics configuration file. It is configured using spark.metrics.conf configuration property. The file is first loaded from the path directly before using Spark’s CLASSPATH.
MetricsConfig accepts a metrics configuration using spark.metrics.conf.-prefixed configuration properties.
Spark comes with conf/metrics.properties.template file that is a template of metrics configuration.
MetricsConfig makes sure that the default metrics properties are always defined.
| Name | Description |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
Note
|
The order of precedence of metrics configuration settings is as follows:
|
MetricsConfig takes a SparkConf when created.
| Name | Description |
|---|---|
|
java.util.Properties with metrics properties Used to initialize per-subsystem’s perInstanceSubProperties. |
|
|
Lookup table of metrics properties per subsystem |
Initializing MetricsConfig — initialize Method
|
1 2 3 4 5 |
initialize(): Unit |
initialize sets the default properties and loads configuration properties from a configuration file (that is defined using spark.metrics.conf configuration property).
initialize takes all Spark properties that start with spark.metrics.conf. prefix from SparkConf and adds them to properties (without the prefix).
In the end, initialize splits configuration per Spark subsystem with the default configuration (denoted as *) assigned to all subsystems afterwards.
|
Note
|
initialize accepts * (star) for the default configuration or any combination of lower- and upper-case letters for Spark subsystem names.
|
|
Note
|
initialize is used exclusively when MetricsSystem is created.
|
setDefaultProperties Internal Method
|
1 2 3 4 5 |
setDefaultProperties(prop: Properties): Unit |
setDefaultProperties sets the default properties (in the input prop).
|
Note
|
setDefaultProperties is used exclusively when MetricsConfig is initialized.
|
Loading Custom Metrics Configuration File or metrics.properties — loadPropertiesFromFile Method
|
1 2 3 4 5 |
loadPropertiesFromFile(path: Option[String]): Unit |
loadPropertiesFromFile tries to open the input path file (if defined) or the default metrics configuration file metrics.properties (on CLASSPATH).
If either file is available, loadPropertiesFromFile loads the properties (to properties registry).
In case of exceptions, you should see the following ERROR message in the logs followed by the exception.
|
1 2 3 4 5 |
ERROR Error loading configuration file [file] |
|
Note
|
loadPropertiesFromFile is used exclusively when MetricsConfig is initialized.
|
Grouping Properties Per Subsystem — subProperties Method
|
1 2 3 4 5 |
subProperties(prop: Properties, regex: Regex): mutable.HashMap[String, Properties] |
subProperties takes prop properties and destructures keys given regex. subProperties takes the matching prefix (of a key per regex) and uses it as a new key with the value(s) being the matching suffix(es).
|
1 2 3 4 5 |
driver.hello.world => (driver, (hello.world)) |
|
Note
|
subProperties is used when MetricsConfig is initialized (to apply the default metrics configuration) and when MetricsSystem registers metrics sources and sinks.
|
