spark技术分享YarnClusterManager — ExternalClusterManager for YARN
YarnClusterManager is the only currently known ExternalClusterManager in Spark. It creates a TaskScheduler and a SchedulerBackend for YARN.
createTaskScheduler Method
createTaskScheduler creates a YarnClusterScheduler for cluster deploy mode and a YarnScheduler for client deploy mode.
It throws a SparkException for unknown deploy modes.
|
1 2 3 4 5 |
Unknown deploy mode '[deployMode]' for Yarn |
createSchedulerBackend Method
createSchedulerBackend creates a YarnClusterSchedulerBackend for cluster deploy mode and a YarnClientSchedulerBackend for client deploy mode.
It throws a SparkException for unknown deploy modes.
|
1 2 3 4 5 |
Unknown deploy mode '[deployMode]' for Yarn |
Initializing YarnClusterManager — initialize Method
initialize simply initializes the input TaskSchedulerImpl.
