DiskBlockObjectWriter-spark技术分享

DiskBlockObjectWriter

DiskBlockObjectWriter is a java.io.OutputStream that BlockManager offers for writing blocks to disk.

Whenever DiskBlockObjectWriter is requested to write a key-value pair, it makes sure that the underlying output streams are open.

DiskBlockObjectWriter can be in the following states (that match the state of the underlying output streams):

Initialized
Open
Closed

Table 1. DiskBlockObjectWriter’s Internal Registries and Counters
Name	Description
`initialized`	Internal flag…FIXME Used when…FIXME
`hasBeenClosed`	Internal flag…FIXME Used when…FIXME
`streamOpen`	Internal flag…FIXME Used when…FIXME
`objOut`	FIXME Used when…FIXME
`mcs`	FIXME Used when…FIXME
`bs`	FIXME Used when…FIXME
`objOut`	FIXME Used when…FIXME
`blockId`	FIXME Used when…FIXME

Note	`DiskBlockObjectWriter` is a `private[spark]` class.

`updateBytesWritten` Method

Caution

FIXME

`initialize` Method

Caution

FIXME

Writing Bytes (From Byte Array Starting From Offset) — `write` Method



write(kvBytes: Array[Byte], offs: Int, len: Int): Unit

write(kvBytes: Array[Byte], offs: Int, len: Int): Unit

write…FIXME

Caution

FIXME

`recordWritten` Method

Caution

FIXME

`commitAndGet` Method



commitAndGet(): FileSegment

commitAndGet(): FileSegment

Note	`commitAndGet` is used when…FIXME

`close` Method

Caution

FIXME

Creating DiskBlockObjectWriter Instance

DiskBlockObjectWriter takes the following when created:

file
serializerManager — SerializerManager
serializerInstance — SerializerInstance
bufferSize
syncWrites flag
writeMetrics — ShuffleWriteMetrics
blockId — BlockId

DiskBlockObjectWriter initializes the internal registries and counters.

Writing Key-Value Pair — `write` Method



write(key: Any, value: Any): Unit

write(key: Any, value: Any): Unit

Before writing, write opens the stream unless already open.

write then writes the key first followed by writing the value.

In the end, write recordWritten.

Note	`write` is used when `BypassMergeSortShuffleWriter` writes records and in `ExternalAppendOnlyMap`, `ExternalSorter` and `WritablePartitionedPairCollection`.

Opening DiskBlockObjectWriter — `open` Method



open(): DiskBlockObjectWriter

open(): DiskBlockObjectWriter

open opens DiskBlockObjectWriter, i.e. initializes and re-sets bs and objOut internal output streams.

Internally, open makes sure that DiskBlockObjectWriter is not closed (i.e. hasBeenClosed flag is disabled). If it was, open throws a IllegalStateException:



Writer already closed. Cannot be reopened.

Writer already closed. Cannot be reopened.

Unless DiskBlockObjectWriter has already been initialized (i.e. initialized flag is enabled), open initializes it (and turns initialized flag on).

Regardless of whether DiskBlockObjectWriter was already initialized or not, open requests SerializerManager to wrap mcs output stream for encryption and compression (for blockId) and sets it as bs.

Note	`open` uses `SerializerManager` that was specified when `DiskBlockObjectWriter` was created

open requests SerializerInstance to serialize bs output stream and sets it as objOut.

Note	`open` uses `SerializerInstance` that was specified when `DiskBlockObjectWriter` was created

In the end, open turns streamOpen flag on.

Note	`open` is used exclusively when `DiskBlockObjectWriter` writes a key-value pair or bytes from a specified byte array but the stream is not open yet.

DiskBlockObjectWriter