spark技术分享 SparkSubmitArguments — spark-submit’s Command-Line Argument Parser
SparkSubmitArguments is a custom SparkSubmitArgumentsParser to handle the command-line arguments of spark-submit script that the actions (i.e. submit, kill and status) use for their execution (possibly with the explicit env environment).
|
Note
|
SparkSubmitArguments is created when launching spark-submit script with only args passed in and later used for printing the arguments in verbose mode.
|
Calculating Spark Properties — loadEnvironmentArguments internal method
|
1 2 3 4 5 |
loadEnvironmentArguments(): Unit |
loadEnvironmentArguments calculates the Spark properties for the current execution of spark-submit.
loadEnvironmentArguments reads command-line options first followed by Spark properties and System’s environment variables.
|
Note
|
Spark config properties start with spark. prefix and can be set using --conf [key=value] command-line option.
|
handle Method
|
1 2 3 4 5 |
protected def handle(opt: String, value: String): Boolean |
handle parses the input opt argument and returns true or throws an IllegalArgumentException when it finds an unknown opt.
handle sets the internal properties in the table Command-Line Options, Spark Properties and Environment Variables.
mergeDefaultSparkProperties Internal Method
|
1 2 3 4 5 |
mergeDefaultSparkProperties(): Unit |
mergeDefaultSparkProperties merges Spark properties from the default Spark properties file, i.e. spark-defaults.conf with those specified through --conf command-line option.
