关注 spark技术分享,
撸spark源码 玩spark最佳实践

SQLMetric — SQL Execution Metric of Physical Operator

SQLMetric — SQL Execution Metric of Physical Operator

SQLMetric is a SQL metric for monitoring execution of a physical operator.

SQLMetric is an accumulator (and that is the mechanism to propagate SQL metric updates on the executors to the driver, e.g. web UI).

Note
Use Details for Query page in SQL tab in web UI to see the SQL execution metrics of a structured query.
Note

SQL metrics are collected using SparkListener. If there are no tasks, Spark SQL cannot collect any metrics. Updates to metrics on the driver-side require explicit call of SQLMetrics.postDriverMetricUpdates.

This is why executing some physical operators (e.g. LocalTableScanExec) may not have SQL metrics in web UI’s Details for Query in SQL tab.

Compare the following SQL queries and their execution pages.


SQLMetric takes a metric type and an initial value when created.

Table 1. Metric Types and Corresponding Create Methods
Metric Type Create Method Failed Values Counted? Description

size

createSizeMetric

no

Used when…​

sum

createMetric

no

Used when…​

timing

createTimingMetric

no

Used when…​

reset Method

reset…​FIXME

Note
reset is used when…​FIXME

Posting Driver-Side Metric Updates — SQLMetrics.postDriverMetricUpdates Method

postDriverMetricUpdates posts a SparkListenerDriverAccumUpdates event to LiveListenerBus when executionId is specified.

Note
postDriverMetricUpdates method belongs to SQLMetrics object.
Note

postDriverMetricUpdates is used when:

赞(0) 打赏
未经允许不得转载:spark技术分享 » SQLMetric — SQL Execution Metric of Physical Operator
分享到: 更多 (0)

关注公众号:spark技术分享

联系我们联系我们

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏