spark技术分享SQLHistoryListener
SQLHistoryListener is a custom SQLListener for History Server. It attaches SQL tab to History Server’s web UI only when the first SparkListenerSQLExecutionStart arrives and shuts onExecutorMetricsUpdate off. It also handles ends of tasks in a slightly different way.
|
Note
|
Support for SQL UI in History Server was added in SPARK-11206 Support SQL UI on the history server. |
|
Caution
|
FIXME Add the link to the JIRA. |
onOtherEvent
|
1 2 3 4 5 |
onOtherEvent(event: SparkListenerEvent): Unit |
When SparkListenerSQLExecutionStart event comes, onOtherEvent attaches SQL tab to web UI and passes the call to the parent SQLListener.
onTaskEnd
|
Caution
|
FIXME |
Creating SQLHistoryListener Instance
SQLHistoryListener is created using a (private[sql]) SQLHistoryListenerFactory class (which is SparkHistoryListenerFactory).
The SQLHistoryListenerFactory class is registered when SparkUI creates a web UI for History Server as a Java service in META-INF/services/org.apache.spark.scheduler.SparkHistoryListenerFactory:
|
1 2 3 4 5 |
org.apache.spark.sql.execution.ui.SQLHistoryListenerFactory |
|
Note
|
Loading the service uses Java’s ServiceLoader.load method. |
