Spark Standalone

Refer to --deploy-mode in spark-submit script.

When you create a SparkContext using spark:// master URL…​FIXME

Keeps track of task ids and executor ids, executors per host, hosts per rack

You can give one or many comma-separated masters URLs in spark:// URL.

A pair of backend and scheduler is returned.

The result is two have a pair of a backend and a scheduler.

Refer to Command-line Options in spark-submit.

If enabled (using spark.deploy.spreadOut), standalone Master attempts to spread out an application’s executors on as many workers as possible (instead of trying to consolidate it onto a small number of nodes).

There is really no need to run multiple workers per machine in Spark 1.5 (perhaps in 1.4, too). You can run multiple executors on the same machine with one worker.

Use SPARK_WORKER_INSTANCES (default: 1) in spark-env.sh to define the number of worker instances.

If you use SPARK_WORKER_INSTANCES, make sure to set SPARK_WORKER_CORES explicitly to limit the cores per worker, or else each worker will try to use all the cores.

You can set up the number of cores as an command line argument when you start a worker daemon using --cores.

Since the change SPARK-1706 Allow multiple executors per worker in Standalone mode in Spark 1.4 it’s currently possible to start multiple executors in a single JVM process of a worker.

To launch multiple executors on a machine you start multiple standalone workers, each with its own JVM. It introduces unnecessary overhead due to these JVM processes, provided that there are enough cores on that worker.

If you are running Spark in standalone mode on memory-rich nodes it can be beneficial to have multiple worker instances on the same node as a very large heap size has two disadvantages:

Mesos and YARN can, out of the box, support packing multiple, smaller executors onto the same physical host, so requesting smaller executors doesn’t mean your application will have fewer overall resources.

SparkDeploySchedulerBackend is the Scheduler Backend for Spark Standalone, i.e. it is used when you create a SparkContext using spark:// master URL.

It requires a Task Scheduler, a Spark context, and a collection of master URLs.

It is a specialized CoarseGrainedSchedulerBackend that uses AppClient and is a AppClientListener.

It uses AppClient to talk to executors.

AppClient is an interface to allow Spark applications to talk to a Standalone cluster (using a RPC Environment). It takes an RPC Environment, a collection of master URLs, a ApplicationDescription, and a AppClientListener.

It is solely used by SparkDeploySchedulerBackend.

AppClient registers AppClient RPC endpoint (using ClientEndpoint class) to a given RPC Environment.

AppClient uses a daemon cached thread pool (askAndReplyThreadPool) with threads’ name in the format of appclient-receive-and-reply-threadpool-ID, where ID is a unique integer for asynchronous asks and replies. It is used for requesting executors (via RequestExecutors message) and kill executors (via KillExecutors).

sendToMaster sends one-way ExecutorStateChanged and UnregisterApplication messages to master.