spark技术分享SparkHadoopUtil
|
Tip
|
Enable Add the following line to
Refer to Logging. |
Creating SparkHadoopUtil Instance — get Method
|
Caution
|
FIXME |
substituteHadoopVariables Method
|
Caution
|
FIXME |
transferCredentials Method
|
Caution
|
FIXME |
newConfiguration Method
|
Caution
|
FIXME |
conf Method
|
Caution
|
FIXME |
stopCredentialUpdater Method
|
Caution
|
FIXME |
Running Executable Block As Spark User — runAsSparkUser Method
|
1 2 3 4 5 |
runAsSparkUser(func: () => Unit) |
runAsSparkUser runs func function with Hadoop’s UserGroupInformation of the current user as a thread local variable (and distributed to child threads). It is later used for authenticating HDFS and YARN calls.
Internally, runAsSparkUser reads the current username (as SPARK_USER environment variable or the short user name from Hadoop’s UserGroupInformation).
|
Caution
|
FIXME How to use SPARK_USER to change the current user name?
|
You should see the current username printed out in the following DEBUG message in the logs:
|
1 2 3 4 5 |
DEBUG YarnSparkHadoopUtil: running as user: [user] |
It then creates a remote user for the current user (using UserGroupInformation.createRemoteUser), transfers credential tokens and runs the input func function as the privileged user.
