RetryingBlockFetcher-spark技术分享

RetryingBlockFetcher

RetryingBlockFetcher is…FIXME

RetryingBlockFetcher is created and immediately started when:

NettyBlockTransferService is requested to fetchBlocks (when maxIORetries is greater than 0 which it is by default)
ExternalShuffleClient is requested to fetchBlocks (when maxIORetries is greater than 0 which it is by default)

RetryingBlockFetcher uses a BlockFetchStarter to createAndStart when requested to start and later initiateRetry.

RetryingBlockFetcher uses outstandingBlocksIds internal registry of outstanding block IDs to fetch that is initially the block IDs to fetch when created.

At initiateRetry, RetryingBlockFetcher prints out the following INFO message to the logs (with the number of outstandingBlocksIds):



Retrying fetch ([retryCount]/[maxRetries]) for [size] outstanding blocks after [retryWaitTime] ms

Retrying fetch ([retryCount]/[maxRetries]) for [size] outstanding blocks after [retryWaitTime] ms

On onBlockFetchSuccess and onBlockFetchFailure, RetryingBlockFetchListener removes the block ID from outstandingBlocksIds.

RetryingBlockFetcher uses a RetryingBlockFetchListener to remove block IDs from the outstandingBlocksIds internal registry.

Creating RetryingBlockFetcher Instance

RetryingBlockFetcher takes the following when created:

Starting RetryingBlockFetcher — `start` Method



void start()

void start()

start simply fetchAllOutstanding.

Note	`start` is used when: `NettyBlockTransferService` is requested to fetchBlocks (when maxIORetries is greater than `0` which it is by default) `ExternalShuffleClient` is requested to fetchBlocks (when maxIORetries is greater than `0` which it is by default)

`initiateRetry` Internal Method



synchronized void initiateRetry()

synchronized void initiateRetry()

initiateRetry…FIXME

Note	`initiateRetry` is used when: `RetryingBlockFetcher` is requested to fetchAllOutstanding `RetryingBlockFetchListener` is requested to onBlockFetchFailure

`fetchAllOutstanding` Internal Method



void fetchAllOutstanding()

void fetchAllOutstanding()

fetchAllOutstanding requests BlockFetchStarter to createAndStart for the outstandingBlocksIds.

Note	`fetchAllOutstanding` is used when `RetryingBlockFetcher` is requested to start and initiateRetry.

RetryingBlockFetchListener

RetryingBlockFetchListener is a BlockFetchingListener that RetryingBlockFetcher uses to remove block IDs from the outstandingBlocksIds internal registry.

`onBlockFetchSuccess` Method



void onBlockFetchSuccess(String blockId, ManagedBuffer data)

void onBlockFetchSuccess(String blockId, ManagedBuffer data)

Note	`onBlockFetchSuccess` is part of BlockFetchingListener Contract to…FIXME.

onBlockFetchSuccess…FIXME

`onBlockFetchFailure` Method



void onBlockFetchFailure(String blockId, Throwable exception)

void onBlockFetchFailure(String blockId, Throwable exception)

Note	`onBlockFetchFailure` is part of BlockFetchingListener Contract to…FIXME.

onBlockFetchFailure…FIXME

RetryingBlockFetcher