spark技术分享YarnRMClient
YarnRMClient is responsible for registering and unregistering a Spark application (in the form of ApplicationMaster) with YARN ResourceManager (and hence RM in the name).
YarnRMClient is just a wrapper for AMRMClient[ContainerRequest] that is started when registering ApplicationMaster (and never stopped explicitly!).
YarnRMClient tracks the application attempt identifiers and the maximum number of attempts to register ApplicationMaster.
| Name | Description |
|---|---|
|
AMRMClient using Created (initialized and started) when Used when |
|
|
Flag to say whether |
|
Tip
|
Enable Add the following line to
Refer to Logging. |
Registering ApplicationMaster with YARN ResourceManager (and Creating YarnAllocator) — register Method
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
register( driverUrl: String, driverRef: RpcEndpointRef, conf: YarnConfiguration, sparkConf: SparkConf, uiAddress: String, uiHistoryAddress: String, securityMgr: SecurityManager, localResources: Map[String, LocalResource]): YarnAllocator |
register creates a AMRMClient, initializes it (using the input YarnConfiguration) and starts immediately.
|
Note
|
AMRMClient is used in YARN to register an application’s ApplicationMaster with the YARN ResourceManager.
|
|
Tip
|
|
register saves the input uiHistoryAddress as uiHistoryAddress.
You should see the following INFO message in the logs (in stderr in YARN):
|
1 2 3 4 5 |
INFO YarnRMClient: Registering the ApplicationMaster |
register then uses AMRMClient to register the Spark application’s ApplicationMaster (using the local hostname, the port 0 and the input uiAddress).
|
Note
|
The input uiAddress is the web UI of the Spark application and is specified using the SparkContext (when the application runs in cluster deploy mode) or using spark.driver.appUIAddress property.
|
registered flag is enabled.
In the end, register creates a new YarnAllocator (using the input parameters of register and the internal AMRMClient).
|
Note
|
register is used exclusively when ApplicationMaster registers itself with the YARN ResourceManager.
|
Unregistering ApplicationMaster from YARN ResourceManager — unregister Method
|
1 2 3 4 5 |
unregister(status: FinalApplicationStatus, diagnostics: String = ""): Unit |
unregister unregisters the ApplicationMaster of a Spark application.
It basically checks that ApplicationMaster is registered and only when it is requests the internal AMRMClient to unregister.
unregister is called when ApplicationMaster wants to unregister.
Maximum Number of Attempts to Register ApplicationMaster — getMaxRegAttempts Method
|
1 2 3 4 5 |
getMaxRegAttempts(sparkConf: SparkConf, yarnConf: YarnConfiguration): Int |
getMaxRegAttempts uses SparkConf and YARN’s YarnConfiguration to read configuration settings and return the maximum number of application attempts before ApplicationMaster registration with YARN is considered unsuccessful (and so the Spark application).
It reads YARN’s yarn.resourcemanager.am.max-attempts (available as YarnConfiguration.RM_AM_MAX_ATTEMPTS) or falls back to YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS (which is 2).
The return value is the minimum of the configuration settings of YARN and Spark.
Getting ApplicationAttemptId of Spark Application — getAttemptId Method
|
1 2 3 4 5 |
getAttemptId(): ApplicationAttemptId |
getAttemptId returns YARN’s ApplicationAttemptId (of the Spark application to which the container was assigned).
Internally, it uses YARN-specific methods like ConverterUtils.toContainerId and ContainerId.getApplicationAttemptId.
getAmIpFilterParams Method
|
Caution
|
FIXME |
