Executors-spark技术分享

Executors Tab

Executors tab in web UI shows…FIXME

Figure 1. Executors Tab in web UI (local mode)

The Executors tab is available under /executors URL, i.e. http://localhost:4040/executors.

Internally, the Executors tab is represented by ExecutorsTab.

Note

What’s interesting in how Storage Memory is displayed in the Executors tab is that the default UnifiedMemoryManager calculates the maximum memory in a way that is different from what the page displays (using the custom JavaScript formatBytes function in utils.js).



// local mode with spark.driver.memory 2g
// ./bin/spark-shell --conf spark.driver.memory=2g
// UnifiedMemoryManager reports 912MB
// You can see it after enabling INFO messages for BlockManagerMasterEndpoint

INFO BlockManagerMasterEndpoint: Registering block manager 192.168.1.8:54503 with 912.3 MB RAM, BlockManagerId(driver, 192.168.1.8, 54503, None)

// custom JavaScript `formatBytes` function (from utils.js) reports...956.6MB
// See https://github.com/apache/spark/blob/master/core/src/main/resources/org/apache/spark/ui/static/utils.js#L40-L48
def formatBytes(bytes: Double) = {
  val k = 1000
  val i = math.floor(math.log(bytes) / math.log(k))
  val maxMemoryWebUI = bytes / math.pow(k, i)
  f"$maxMemoryWebUI%1.1f"
}
scala> println(formatBytes(maxMemory))
956.6

// local mode with spark.driver.memory 2g

// ./bin/spark-shell --conf spark.driver.memory=2g

// UnifiedMemoryManager reports 912MB

// You can see it after enabling INFO messages for BlockManagerMasterEndpoint

INFO BlockManagerMasterEndpoint: Registering block manager 192.168.1.8:54503 with 912.3 MB RAM, BlockManagerId(driver, 192.168.1.8, 54503, None)

// custom JavaScript `formatBytes` function (from utils.js) reports...956.6MB

// See https://github.com/apache/spark/blob/master/core/src/main/resources/org/apache/spark/ui/static/utils.js#L40-L48

def formatBytes(bytes: Double) = {

val k = 1000

val i = math.floor(math.log(bytes) / math.log(k))

val maxMemoryWebUI = bytes / math.pow(k, i)

f"$maxMemoryWebUI%1.1f"

}

scala> println(formatBytes(maxMemory))

956.6

`getExecInfo` Method



getExecInfo(
  listener: ExecutorsListener,
  statusId: Int,
  isActive: Boolean): ExecutorSummary

getExecInfo(

listener: ExecutorsListener,

statusId: Int,

isActive: Boolean): ExecutorSummary

getExecInfo creates a ExecutorSummary.

Caution

FIXME

Note	`getExecInfo` is used when…FIXME

Settings

spark.ui.threadDumpsEnabled

spark.ui.threadDumpsEnabled (default: true) is to enable (true) or disable (false) ExecutorThreadDumpPage.

Executors

Executors Tab

`getExecInfo` Method

Settings

spark.ui.threadDumpsEnabled

相关推荐

欢迎关注：spark技术分享

热门标签

近期文章

分类目录

关注公众号：spark技术分享

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏

QQ咨询

回顶部

Executors Tab

getExecInfo Method

Settings

spark.ui.threadDumpsEnabled

相关推荐

欢迎关注：spark技术分享

热门标签

近期文章

分类目录

关注公众号：spark技术分享

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏

QQ咨询

回顶部

`getExecInfo` Method