StateStoreCoordinatorRef — RPC Endpoint Reference to StateStoreCoordinator-spark技术分享

StateStoreCoordinatorRef — RPC Endpoint Reference to StateStoreCoordinator

StateStoreCoordinatorRef is used to (let the tasks on Spark executors to) send messages to the StateStoreCoordinator (that lives on the driver).

StateStoreCoordinatorRef is given the RpcEndpointRef to the StateStoreCoordinator RPC endpoint when created.

StateStoreCoordinatorRef is created through StateStoreCoordinatorRef helper object when requested to create one for the driver (when StreamingQueryManager is created) or an executor (when StateStore helper object is requested for the RPC endpoint reference to StateStoreCoordinator for Executors).

Method Description

deactivateInstances



deactivateInstances(runId: UUID): Unit

deactivateInstances(runId: UUID): Unit

Requests the RpcEndpointRef to send a DeactivateInstances synchronous message with the given runId and waits for a true / false response

Used exclusively when StreamingQueryManager is requested to handle termination of a streaming query (when StreamExecution is requested to run a streaming query and the query has finished (running streaming batches)).

getLocation



getLocation(
  stateStoreProviderId: StateStoreProviderId): Option[String]

getLocation(

stateStoreProviderId: StateStoreProviderId): Option[String]

Requests the RpcEndpointRef to send a GetLocation synchronous message with the given StateStoreProviderId and waits for the location

Used when:

StateStoreAwareZipPartitionsRDD is requested to getPreferredLocations (when StreamingSymmetricHashJoinExec physical operator is requested to doExecute)
StateStoreRDD is requested for preferred locations for a task for a partition

reportActiveInstance



reportActiveInstance(
  stateStoreProviderId: StateStoreProviderId,
  host: String,
  executorId: String): Unit

reportActiveInstance(

stateStoreProviderId: StateStoreProviderId,

host: String,

executorId: String): Unit

Requests the RpcEndpointRef to send a ReportActiveInstance one-way asynchronous (fire-and-forget) message with the given StateStoreProviderId, host and executorId

Used exclusively when StateStore helper object is requested for reportActiveStoreInstance (when StateStore helper object is requested to find the StateStore by StateStoreProviderId)

stop



stop(): Unit

stop(): Unit

Requests the RpcEndpointRef to send a StopCoordinator synchronous message

Used exclusively for unit testing

verifyIfInstanceActive



verifyIfInstanceActive(
  stateStoreProviderId: StateStoreProviderId,
  executorId: String): Boolean

verifyIfInstanceActive(

stateStoreProviderId: StateStoreProviderId,

executorId: String): Boolean

Requests the RpcEndpointRef to send a VerifyIfInstanceActive synchronous message with the given StateStoreProviderId and executorId, and waits for a true / false response

Used exclusively when StateStore helper object is requested for verifyIfStoreInstanceActive (when requested to doMaintenance from a running MaintenanceTask daemon thread)

Creating StateStoreCoordinatorRef to StateStoreCoordinator RPC Endpoint for Driver — `forDriver` Factory Method



forDriver(env: SparkEnv): StateStoreCoordinatorRef

forDriver(env: SparkEnv): StateStoreCoordinatorRef

forDriver…FIXME

Note	`forDriver` is used exclusively when `StreamingQueryManager` is created.

Creating StateStoreCoordinatorRef to StateStoreCoordinator RPC Endpoint for Executor — `forExecutor` Factory Method



forExecutor(env: SparkEnv): StateStoreCoordinatorRef

forExecutor(env: SparkEnv): StateStoreCoordinatorRef

forExecutor…FIXME

Note	`forExecutor` is used exclusively when `StateStore` helper object is requested for the RPC endpoint reference to StateStoreCoordinator for Executors.

StateStoreCoordinatorRef — RPC Endpoint Reference to StateStoreCoordinator