ExecutorAllocationClient-spark技术分享

ExecutorAllocationClient

ExecutorAllocationClient is a contract for clients to communicate with a cluster manager to request or kill executors.

ExecutorAllocationClient Contract



trait ExecutorAllocationClient {
  def getExecutorIds(): Seq[String]
  def requestTotalExecutors(numExecutors: Int, localityAwareTasks: Int, hostToLocalTaskCount: Map[String, Int]): Boolean
  def requestExecutors(numAdditionalExecutors: Int): Boolean
  def killExecutor(executorId: String): Boolean
  def killExecutors(executorIds: Seq[String]): Seq[String]
  def killExecutorsOnHost(host: String): Boolean
}

trait ExecutorAllocationClient {

def getExecutorIds(): Seq[String]

def requestTotalExecutors(numExecutors: Int, localityAwareTasks: Int, hostToLocalTaskCount: Map[String, Int]): Boolean

def requestExecutors(numAdditionalExecutors: Int): Boolean

def killExecutor(executorId: String): Boolean

def killExecutors(executorIds: Seq[String]): Seq[String]

def killExecutorsOnHost(host: String): Boolean

}

Note	`ExecutorAllocationClient` is a `private[spark]` contract.

Table 1. ExecutorAllocationClient Contract
Method	Description
`getExecutorIds`	Finds identifiers of the executors in use. Used when `SparkContext` calculates the executors in use and also when Spark Streaming manages executors.
`requestTotalExecutors`	Updates the cluster manager with the exact number of executors desired. It returns whether the request has been acknowledged by the cluster manager (`true`) or not (`false`). Used when: `SparkContext` requests executors (for coarse-grained scheduler backends only). `ExecutorAllocationManager` starts, does updateAndSyncNumExecutorsTarget, and addExecutors. Streaming `ExecutorAllocationManager` requests executors. `YarnSchedulerBackend` stops.
`requestExecutors`	Requests additional executors from a cluster manager and returns whether the request has been acknowledged by the cluster manager (`true`) or not (`false`). Used when `SparkContext` requests additional executors (for coarse-grained scheduler backends only).
`killExecutor`	Requests a cluster manager to kill a single executor that is no longer in use and returns whether the request has been acknowledged by the cluster manager (`true`) or not (`false`). The default implementation simply calls killExecutors (with a single-element collection of executors to kill). Used when: `ExecutorAllocationManager` removes an executor. `SparkContext` is requested to kill executors. Streaming `ExecutorAllocationManager` is requested to kill executors.
`killExecutors`	Requests that a cluster manager to kill one or many executors that are no longer in use and returns whether the request has been acknowledged by the cluster manager (`true`) or not (`false`). Interestingly, it is only used for killExecutor.
`killExecutorsOnHost`	Used exclusively when `BlacklistTracker` kills blacklisted executors.

ExecutorAllocationClient

ExecutorAllocationClient

ExecutorAllocationClient Contract

相关推荐

欢迎关注：spark技术分享

热门标签

近期文章

分类目录

关注公众号：spark技术分享

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏

QQ咨询

回顶部