spark技术分享Local Properties — Creating Logical Job Groups
The purpose of local properties concept is to create logical groups of jobs by means of properties that (regardless of the threads used to submit the jobs) makes the separate jobs launched from different threads belong to a single logical group.
You can set a local property that will affect Spark jobs submitted from a thread, such as the Spark fair scheduler pool. You can use your own custom properties. The properties are propagated through to worker tasks and can be accessed there via TaskContext.getLocalProperty.
|
Note
|
Propagating local properties to workers starts when SparkContext is requested to run or submit a Spark job that in turn passes them along to DAGScheduler.
|
|
Note
|
Local properties is used to group jobs into pools in FAIR job scheduler by spark.scheduler.pool per-thread property and in SQLExecution.withNewExecutionId Helper Methods |
A common use case for the local property concept is to set a local property in a thread, say spark.scheduler.pool, after which all jobs submitted within the thread will be grouped, say into a pool by FAIR job scheduler.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
val rdd = sc.parallelize(0 to 9) sc.setLocalProperty("spark.scheduler.pool", "myPool") // these two jobs (one per action) will run in the myPool pool rdd.count rdd.collect sc.setLocalProperty("spark.scheduler.pool", null) // this job will run in the default pool rdd.count |
Local Properties — localProperties Property
|
1 2 3 4 5 |
localProperties: InheritableThreadLocal[Properties] |
localProperties is a protected[spark] property of a SparkContext that are the properties through which you can create logical job groups.
|
Tip
|
Read up on Java’s java.lang.InheritableThreadLocal. |
Setting Local Property — setLocalProperty Method
|
1 2 3 4 5 |
setLocalProperty(key: String, value: String): Unit |
setLocalProperty sets key local property to value.
|
Tip
|
When value is null the key property is removed from localProperties.
|
Getting Local Property — getLocalProperty Method
|
1 2 3 4 5 |
getLocalProperty(key: String): String |
getLocalProperty gets a local property by key in this thread. It returns null if key is missing.
Getting Local Properties — getLocalProperties Method
|
1 2 3 4 5 |
getLocalProperties: Properties |
getLocalProperties is a private[spark] method that gives access to localProperties.
setLocalProperties Method
|
1 2 3 4 5 |
setLocalProperties(props: Properties): Unit |
setLocalProperties is a private[spark] method that sets props as localProperties.
