spark技术分享MemoryConsumer
MemoryConsumer is the contract for memory consumers of TaskMemoryManager with support for spilling.
A MemoryConsumer basically tracks how much memory is allocated.
Creating a MemoryConsumer requires a TaskMemoryManager with optional pageSize and a MemoryMode.
|
Note
|
If not specified, pageSize defaults to TaskMemoryManager.pageSizeBytes and ON_HEAP memory mode.
|
spill Method
|
1 2 3 4 5 6 |
abstract long spill(long size, MemoryConsumer trigger) throws IOException |
|
Caution
|
FIXME |
Memory Allocated — used Registry
used is the amount of memory in use (i.e. allocated) by the MemoryConsumer.
Deallocate MemoryBlock — freePage Method
|
1 2 3 4 5 |
protected void freePage(MemoryBlock page) |
freePage is a protected method to deallocate the MemoryBlock.
Internally, it decrements used registry by the size of page and frees the page.
Allocate LongArray — allocateArray Method
|
1 2 3 4 5 |
LongArray allocateArray(long size) |
allocateArray allocates LongArray of size length.
Internally, it allocates a page for the requested size. The size is recorded in the internal used counter.
However, if it was not possible to allocate the size memory, it shows the current memory usage and a OutOfMemoryError is thrown.
|
1 2 3 4 5 |
Unable to acquire [required] bytes of memory, got [got] |
Acquiring Memory — acquireMemory Method
|
1 2 3 4 5 |
long acquireMemory(long size) |
acquireMemory acquires execution memory of size size. The memory is recorded in used registry.
throwOom Internal Method
|
1 2 3 4 5 |
void throwOom(final MemoryBlock page, final long required) |
throwOom…FIXME
|
Note
|
throwOom is used when MemoryConsumer is requested to allocateArray and allocatePage.
|
