spark技术分享YarnSparkHadoopUtil
YarnSparkHadoopUtil is…FIXME
YarnSparkHadoopUtil can only be created when SPARK_YARN_MODE flag is enabled.
|
Note
|
YarnSparkHadoopUtil belongs to org.apache.spark.deploy.yarn package.
|
|
Tip
|
Enable Add the following line to
Refer to Logging. |
startCredentialUpdater Method
|
Caution
|
FIXME |
Getting YarnSparkHadoopUtil Instance — get Method
|
Caution
|
FIXME |
addPathToEnvironment Method
|
1 2 3 4 5 |
addPathToEnvironment(env: HashMap[String, String], key: String, value: String): Unit |
|
Caution
|
FIXME |
startExecutorDelegationTokenRenewer
|
Caution
|
FIXME |
stopExecutorDelegationTokenRenewer
|
Caution
|
FIXME |
getApplicationAclsForYarn Method
|
Caution
|
FIXME |
Resolving Environment Variable — expandEnvironment Method
|
1 2 3 4 5 |
expandEnvironment(environment: Environment): String |
expandEnvironment resolves environment variable using YARN’s Environment.$ or Environment.$$ methods (depending on the version of Hadoop used).
Computing YARN’s ContainerId — getContainerId Method
|
1 2 3 4 5 |
getContainerId: ContainerId |
getContainerId is a private[spark] method that gets YARN’s ContainerId from the YARN environment variable ApplicationConstants.Environment.CONTAINER_ID and converts it to the return object using YARN’s ConverterUtils.toContainerId.
Calculating Initial Number of Executors — getInitialTargetExecutorNumber Method
|
1 2 3 4 5 |
getInitialTargetExecutorNumber(conf: SparkConf, numExecutors: Int = 2): Int |
getInitialTargetExecutorNumber calculates the initial number of executors for Spark on YARN. It varies by whether dynamic allocation is enabled or not.
|
Note
|
The default number of executors (aka DEFAULT_NUMBER_EXECUTORS) is 2.
|
With dynamic allocation enabled, getInitialTargetExecutorNumber is spark.dynamicAllocation.initialExecutors or spark.dynamicAllocation.minExecutors to fall back to 0 if the others are undefined.
With dynamic allocation disabled, getInitialTargetExecutorNumber is the value of spark.executor.instances property or SPARK_EXECUTOR_INSTANCES environment variable, or the default value (of the input parameter numExecutors) 2.
|
Note
|
getInitialTargetExecutorNumber is used to calculate totalExpectedExecutors to start Spark on YARN in client or cluster modes.
|
