spark技术分享MemoryManager — Memory Management System
MemoryManager is the base of memory managers that manage shared memory for task execution and block storage.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
package org.apache.spark.memory abstract class MemoryManager(...) { // only required methods that have no implementation // the others follow def acquireExecutionMemory( numBytes: Long, taskAttemptId: Long, memoryMode: MemoryMode): Long def acquireStorageMemory(blockId: BlockId, numBytes: Long, memoryMode: MemoryMode): Boolean def acquireUnrollMemory(blockId: BlockId, numBytes: Long, memoryMode: MemoryMode): Boolean def maxOffHeapStorageMemory: Long def maxOnHeapStorageMemory: Long } |
|
Note
|
MemoryManager is a private[spark] contract.
|
| Method | Description |
|---|---|
|
|
Used exclusively when |
|
|
|
|
|
Used exclusively when |
|
|
|
|
|
|
|
Note
|
MemoryManager is a Scala abstract class and cannot be created directly, but only as one of the implementations.
|
Execution memory is used for computation in shuffles, joins, sorts and aggregations.
Storage memory is used for caching and propagating internal data across the nodes in a cluster.
| MemoryManager | Description |
|---|---|
| Name | Description | ||
|---|---|---|---|
|
|
Used when…FIXME |
||
|
|
Used when…FIXME |
||
|
|
Used when…FIXME |
||
|
|
Used when…FIXME
|
Creating MemoryManager Instance
MemoryManager takes the following when created:
MemoryManager initializes the internal registries and counters.
|
Note
|
MemoryManager is a Scala abstract class and cannot be created directly, but only as one of the implementations.
|
releaseExecutionMemory Method
|
1 2 3 4 5 6 7 8 |
releaseExecutionMemory( numBytes: Long, taskAttemptId: Long, memoryMode: MemoryMode): Unit |
releaseExecutionMemory…FIXME
|
Note
|
releaseExecutionMemory is used when TaskMemoryManager is requested to releaseExecutionMemory and cleanUpAllAllocatedMemory
|
releaseAllExecutionMemoryForTask Method
|
1 2 3 4 5 |
releaseAllExecutionMemoryForTask(taskAttemptId: Long): Long |
releaseAllExecutionMemoryForTask…FIXME
|
Note
|
releaseAllExecutionMemoryForTask is used exclusively when TaskRunner is requested to run (and cleans up after itself).
|
tungstenMemoryMode Flag
|
1 2 3 4 5 |
tungstenMemoryMode: MemoryMode |
tungstenMemoryMode returns OFF_HEAP only when the following are all met:
-
spark.memory.offHeap.enabled configuration property is enabled (it is not by default)
-
spark.memory.offHeap.size configuration property is greater than
0(it is0by default) -
JVM supports unaligned memory access (aka unaligned Unsafe, i.e.
sun.misc.Unsafepackage is available and the underlying system has unaligned-access capability)
Otherwise, tungstenMemoryMode returns ON_HEAP.
|
Note
|
Given that spark.memory.offHeap.enabled configuration property is disabled (false) by default and spark.memory.offHeap.size configuration property is 0 by default, Spark seems to encourage using Tungsten memory allocated on the JVM heap (ON_HEAP).
|
|
Note
|
tungstenMemoryMode is a Scala final val and cannot be changed by custom MemoryManagers.
|
|
Note
|
|
