spark技术分享AbstractCommandBuilder
AbstractCommandBuilder is the base command builder for SparkSubmitCommandBuilder and SparkClassCommandBuilder specialized command builders.
AbstractCommandBuilder expects that command builders define buildCommand.
| Method | Description |
|---|---|
|
|
The only abstract method that subclasses have to define. |
|
Loads the configuration file for a Spark application, be it the user-specified properties file or |
buildJavaCommand Internal Method
|
1 2 3 4 5 |
List<String> buildJavaCommand(String extraClassPath) |
buildJavaCommand builds the Java command for a Spark application (which is a collection of elements with the path to java executable, JVM options from java-opts file, and a class path).
If javaHome is set, buildJavaCommand adds [javaHome]/bin/java to the result Java command. Otherwise, it uses JAVA_HOME or, when no earlier checks succeeded, falls through to java.home Java’s system property.
|
Caution
|
FIXME Who sets javaHome internal property and when?
|
buildJavaCommand loads extra Java options from the java-opts file in configuration directory if the file exists and adds them to the result Java command.
Eventually, buildJavaCommand builds the class path (with the extra class path if non-empty) and adds it as -cp to the result Java command.
buildClassPath method
|
1 2 3 4 5 |
List<String> buildClassPath(String appClassPath) |
buildClassPath builds the classpath for a Spark application.
|
Note
|
Directories always end up with the OS-specific file separator at the end of their paths. |
buildClassPath adds the following in that order:
-
SPARK_CLASSPATHenvironment variable -
The input
appClassPath -
(only with
SPARK_PREPEND_CLASSESset orSPARK_TESTINGbeing1) Locally compiled Spark classes inclasses,test-classesand Core’s jars.CautionFIXME Elaborate on “locally compiled Spark classes”. -
(only with
SPARK_SQL_TESTINGbeing1) …CautionFIXME Elaborate on the SQL testing case -
HADOOP_CONF_DIRenvironment variable -
YARN_CONF_DIRenvironment variable -
SPARK_DIST_CLASSPATHenvironment variable
|
Note
|
childEnv is queried first before System properties. It is always empty for AbstractCommandBuilder (and SparkSubmitCommandBuilder, too).
|
Loading Properties File — loadPropertiesFile Internal Method
|
1 2 3 4 5 |
Properties loadPropertiesFile() |
loadPropertiesFile is part of AbstractCommandBuilder private API that loads Spark settings from a properties file (when specified on the command line) or spark-defaults.conf in the configuration directory.
It loads the settings from the following files starting from the first and checking every location until the first properties file is found:
-
propertiesFile(if specified using--properties-filecommand-line option or set byAbstractCommandBuilder.setPropertiesFile). -
[SPARK_CONF_DIR]/spark-defaults.conf -
[SPARK_HOME]/conf/spark-defaults.conf
|
Note
|
loadPropertiesFile reads a properties file using UTF-8.
|
Spark’s Configuration Directory — getConfDir Internal Method
AbstractCommandBuilder uses getConfDir to compute the current configuration directory of a Spark application.
It uses SPARK_CONF_DIR (from childEnv which is always empty anyway or as a environment variable) and falls through to [SPARK_HOME]/conf (with SPARK_HOME from getSparkHome internal method).
Spark’s Home Directory — getSparkHome Internal Method
AbstractCommandBuilder uses getSparkHome to compute Spark’s home directory for a Spark application.
It uses SPARK_HOME (from childEnv which is always empty anyway or as a environment variable).
If SPARK_HOME is not set, Spark throws a IllegalStateException:
|
1 2 3 4 5 |
Spark home not found; set it explicitly or use the SPARK_HOME environment variable. |
