spark技术分享SparkSqlParser — Default SQL Parser
SparkSqlParser is the default SQL parser of the SQL statements supported in Spark SQL.
SparkSqlParser supports variable substitution.
SparkSqlParser uses SparkSqlAstBuilder (as AstBuilder).
|
Note
|
Spark SQL supports SQL statements as described in SqlBase.g4 ANTLR grammar. |
SparkSqlParser is available as sqlParser of a SessionState.
|
1 2 3 4 5 6 |
val spark: SparkSession = ... spark.sessionState.sqlParser |
SparkSqlParser is used to translate an expression to the corresponding Column in the following:
-
expr function
-
Dataset.selectExpr operator
-
Dataset.filter operator
-
Dataset.where operator
|
1 2 3 4 5 6 7 |
scala> expr("token = 'hello'") 16/07/07 18:32:53 INFO SparkSqlParser: Parsing command: token = 'hello' res0: org.apache.spark.sql.Column = (token = hello) |
SparkSqlParser is used to parse table strings into their corresponding table identifiers in the following:
-
tablemethods in DataFrameReader and SparkSession -
insertInto and saveAsTable methods of
DataFrameWriter -
createExternalTableandrefreshTablemethods of Catalog (and SessionState)
SparkSqlParser is used to translate a SQL text to its corresponding logical operator in SparkSession.sql method.
|
Tip
|
Enable Add the following line to
Refer to Logging. |
