Spark on YARN

You can submit Spark applications to a Hadoop YARN cluster using yarn master URL.

Without specifying the deploy mode, it is assumed client.

There are two deploy modes for YARN — client (default) or cluster.

In client mode the Spark driver (and SparkContext) runs on a client node outside a YARN cluster whereas in cluster mode it runs inside a YARN cluster, i.e. inside a YARN container alongside ApplicationMaster (that acts as the Spark application in YARN).

In that sense, a Spark application deployed to YARN is a YARN-compatible execution framework that can be deployed to a YARN cluster (alongside other Hadoop workloads). On YARN, a Spark executor maps to a single YARN container.

Spark on YARN supports multiple application attempts and supports data locality for data in HDFS. You can also take advantage of Hadoop’s security and run Spark in a secure Hadoop environment using Kerberos authentication (aka Kerberized clusters).

There are few settings that are specific to YARN (see Settings). Among them, you can particularly like the support for YARN resource queues (to divide cluster resources and allocate shares to different teams and users based on advanced policies).

The memory in the YARN resource requests is --executor-memory + what’s set for spark.yarn.executor.memoryOverhead, which defaults to 10% of --executor-memory.

If YARN has enough resources it will deploy the executors distributed across the cluster, then each of them will try to process the data locally (NODE_LOCAL in Spark Web UI), with as many splits in parallel as you defined in spark.executor.cores.