BlockManagerSource — Metrics Source for BlockManager-spark技术分享

BlockManagerSource — Metrics Source for BlockManager

BlockManagerSource is the metrics source of a BlockManager.

BlockManagerSource is registered under the name BlockManager (when SparkContext is created).

Table 1. BlockManagerSource’s Gauge Metrics (in alphabetical order)
Name	Type	Description
`disk.diskSpaceUsed_MB`	long	Requests `BlockManagerMaster` for storage status (for every BlockManager) and sums up their disk space used (`diskUsed`).
`memory.maxMem_MB`	long	Requests `BlockManagerMaster` for storage status (for every BlockManager) and sums up their maximum memory limit (`maxMem`).
`memory.maxOffHeapMem_MB`	long	Requests `BlockManagerMaster` for storage status (for every BlockManager) and sums up their off-heap memory remaining (`maxOffHeapMem`).
`memory.maxOnHeapMem_MB`	long	Requests `BlockManagerMaster` for storage status (for every BlockManager) and sums up their on-heap memory remaining (`maxOnHeapMem`).
`memory.memUsed_MB`	long	Requests `BlockManagerMaster` for storage status (for every BlockManager) and sums up their memory used (`memUsed`).
`memory.offHeapMemUsed_MB`	long	Requests `BlockManagerMaster` for storage status (for every BlockManager) and sums up their off-heap memory used (`offHeapMemUsed`).
`memory.onHeapMemUsed_MB`	long	Requests `BlockManagerMaster` for storage status (for every BlockManager) and sums up their on-heap memory used (`onHeapMemUsed`).
`memory.remainingMem_MB`	long	Requests `BlockManagerMaster` for storage status (for every BlockManager) and sums up their memory remaining (`memRemaining`).
`memory.remainingOffHeapMem_MB`	long	Requests `BlockManagerMaster` for storage status (for every BlockManager) and sums up their off-heap memory remaining (`offHeapMemRemaining`).
`memory.remainingOnHeapMem_MB`	long	Requests `BlockManagerMaster` for storage status (for every BlockManager) and sums up their on-heap memory remaining (`onHeapMemRemaining`).

You can access the BlockManagerSource metrics using the web UI’s port (as spark.ui.port configuration property).



$ http --follow http://localhost:4040/metrics/json \
    | jq '.gauges | keys | .[] | select(test(".driver.BlockManager"; "g"))'
"local-1528725411625.driver.BlockManager.disk.diskSpaceUsed_MB"
"local-1528725411625.driver.BlockManager.memory.maxMem_MB"
"local-1528725411625.driver.BlockManager.memory.maxOffHeapMem_MB"
"local-1528725411625.driver.BlockManager.memory.maxOnHeapMem_MB"
"local-1528725411625.driver.BlockManager.memory.memUsed_MB"
"local-1528725411625.driver.BlockManager.memory.offHeapMemUsed_MB"
"local-1528725411625.driver.BlockManager.memory.onHeapMemUsed_MB"
"local-1528725411625.driver.BlockManager.memory.remainingMem_MB"
"local-1528725411625.driver.BlockManager.memory.remainingOffHeapMem_MB"
"local-1528725411625.driver.BlockManager.memory.remainingOnHeapMem_MB"

$ http --follow http://localhost:4040/metrics/json \

| jq '.gauges | keys | .[] | select(test(".driver.BlockManager"; "g"))'

"local-1528725411625.driver.BlockManager.disk.diskSpaceUsed_MB"

"local-1528725411625.driver.BlockManager.memory.maxMem_MB"

"local-1528725411625.driver.BlockManager.memory.maxOffHeapMem_MB"

"local-1528725411625.driver.BlockManager.memory.maxOnHeapMem_MB"

"local-1528725411625.driver.BlockManager.memory.memUsed_MB"

"local-1528725411625.driver.BlockManager.memory.offHeapMemUsed_MB"

"local-1528725411625.driver.BlockManager.memory.onHeapMemUsed_MB"

"local-1528725411625.driver.BlockManager.memory.remainingMem_MB"

"local-1528725411625.driver.BlockManager.memory.remainingOffHeapMem_MB"

"local-1528725411625.driver.BlockManager.memory.remainingOnHeapMem_MB"

BlockManagerSource takes a BlockManager when created.

BlockManagerSource is created when SparkContext is created.

BlockManagerSource — Metrics Source for BlockManager

BlockManagerSource — Metrics Source for BlockManager

相关推荐

欢迎关注：spark技术分享

热门标签

近期文章

分类目录

关注公众号：spark技术分享

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏

QQ咨询

回顶部