SQLMetric — SQL Execution Metric of Physical Operator-spark技术分享

SQLMetric — SQL Execution Metric of Physical Operator

SQLMetric is a SQL metric for monitoring execution of a physical operator.

SQLMetric is an accumulator (and that is the mechanism to propagate SQL metric updates on the executors to the driver, e.g. web UI).

Note	Use Details for Query page in SQL tab in web UI to see the SQL execution metrics of a structured query.

Note

SQL metrics are collected using SparkListener. If there are no tasks, Spark SQL cannot collect any metrics. Updates to metrics on the driver-side require explicit call of SQLMetrics.postDriverMetricUpdates.

This is why executing some physical operators (e.g. LocalTableScanExec) may not have SQL metrics in web UI’s Details for Query in SQL tab.

Compare the following SQL queries and their execution pages.



// The query does not have SQL metrics in web UI
Seq("Jacek").toDF("name").show

// The query gives numOutputRows metric in web UI's Details for Query (SQL tab)
Seq("Jacek").toDF("name").count

// The query does not have SQL metrics in web UI

Seq("Jacek").toDF("name").show

// The query gives numOutputRows metric in web UI's Details for Query (SQL tab)

Seq("Jacek").toDF("name").count

SQLMetric takes a metric type and an initial value when created.

Table 1. Metric Types and Corresponding Create Methods
Metric Type	Create Method	Failed Values Counted?	Description
`size`	`createSizeMetric`	no	Used when…
`sum`	`createMetric`	no	Used when…
`timing`	`createTimingMetric`	no	Used when…

`reset` Method



reset(): Unit

reset(): Unit

reset…FIXME

Note	`reset` is used when…FIXME

Posting Driver-Side Metric Updates — `SQLMetrics.postDriverMetricUpdates` Method



postDriverMetricUpdates(
  sc: SparkContext,
  executionId: String,
  metrics: Seq[SQLMetric]): Unit

postDriverMetricUpdates(

sc: SparkContext,

executionId: String,

metrics: Seq[SQLMetric]): Unit

postDriverMetricUpdates posts a SparkListenerDriverAccumUpdates event to LiveListenerBus when executionId is specified.

Note	`postDriverMetricUpdates` method belongs to `SQLMetrics` object.

Note

postDriverMetricUpdates is used when:

BroadcastExchangeExec is requested to prepare for execution (and initializes relationFuture for the first time)
FileSourceScanExec physical operator is requested for selectedPartitions (and posts updates to numFiles and metadataTime metrics)
SubqueryExec physical operator is requested to prepare for execution (and initializes relationFuture for the first time that in turn posts updates to collectTime and dataSize metrics)

SQLMetric — SQL Execution Metric of Physical Operator