spark技术分享YarnClusterScheduler — TaskScheduler for Cluster Deploy Mode
YarnClusterScheduler is the TaskScheduler for Spark on YARN in cluster deploy mode.
It is a custom YarnScheduler that makes sure that appropriate initialization of ApplicationMaster is performed, i.e. SparkContext is initialized and stopped.
While being created, you should see the following INFO message in the logs:
|
1 2 3 4 5 |
INFO YarnClusterScheduler: Created YarnClusterScheduler |
|
Tip
|
Enable Add the following line to
Refer to Logging. |
postStartHook Callback
postStartHook calls ApplicationMaster.sparkContextInitialized before the parent’s postStartHook.
You should see the following INFO message in the logs:
|
1 2 3 4 5 |
INFO YarnClusterScheduler: YarnClusterScheduler.postStartHook done |
Stopping YarnClusterScheduler (stop method)
stop calls the parent’s stop followed by ApplicationMaster.sparkContextStopped.
