spark技术分享MemoryManager Configuration Properties
The page describes the memory-related configuration properties of Apache Spark (mostly MemoryManager and “relatives”).
|
Note
|
It may be that the page should be somewhere else. Any suggestions? |
Use the following idiom to set the configuration properties.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
import org.apache.spark.internal.config._ import org.apache.spark.SparkConf // all properties are private[spark], sorry // the following won't work unless the code is under org.apache.spark package val sparkConf = new SparkConf().set(MEMORY_OFFHEAP_ENABLED.key, "false") import org.apache.spark.SparkConf val sparkConf = new SparkConf() scala> println(sparkConf.getOption("spark.memory.offHeap.enabled")) None // Using ConfigEntry-based SparkConf.set that is also private[spark] // Use :pa -raw in spark-shell to paste the following snippet // BEGIN package org.apache.spark.japila object Config { import org.apache.spark.SparkConf import org.apache.spark.internal.config._ def with_MEMORY_OFFHEAP_ENABLED(sparkConf: SparkConf = new SparkConf()): SparkConf = { sparkConf.set(MEMORY_OFFHEAP_ENABLED, true) } } // END import org.apache.spark.japila.Config val sparkConf_OffHeap = Config.with_MEMORY_OFFHEAP_ENABLED(sparkConf) scala> println(sparkConf_OffHeap.get("spark.memory.offHeap.enabled")) true |
| Name | Default Value | Description | ||
|---|---|---|---|---|
|
|
|
Controls whether Spark will attempt to use off-heap memory for certain operations ( Tracks whether Tungsten memory will be allocated on the JVM heap or off-heap (using If enabled, spark.memory.offHeap.size has to be greater than 0. Used exclusively when
|
||
|
|
