AbstractSqlParser — Base SQL Parsing Infrastructure-spark技术分享

AbstractSqlParser — Base SQL Parsing Infrastructure

AbstractSqlParser is the base of ParserInterfaces that use an AstBuilder to parse SQL statements and convert them to Spark SQL entities, i.e. DataType, StructType, Expression, LogicalPlan and TableIdentifier.

AbstractSqlParser is the foundation of the SQL parsing infrastructure.



package org.apache.spark.sql.catalyst.parser

abstract class AbstractSqlParser extends ParserInterface {
  // only required properties (vals and methods) that have no implementation
  // the others follow
  def astBuilder: AstBuilder
}

package org.apache.spark.sql.catalyst.parser

abstract class AbstractSqlParser extends ParserInterface {

// only required properties (vals and methods) that have no implementation

// the others follow

def astBuilder: AstBuilder

}

Table 1. AbstractSqlParser Contract
Method	Description
`astBuilder`	AstBuilder for parsing SQL statements. Used in all the `parse` methods, i.e. parseDataType, parseExpression, parsePlan, parseTableIdentifier, and parseTableSchema.

Name Description

SparkSqlParser

The default SQL parser in SessionState available as sqlParser property.



val spark: SparkSession = ...
spark.sessionState.sqlParser

val spark: SparkSession = ...

spark.sessionState.sqlParser

CatalystSqlParser

Creates a DataType or a StructType (schema) from their canonical string representation.

Setting Up SqlBaseLexer and SqlBaseParser for Parsing — `parse` Method



parse[T](command: String)(toResult: SqlBaseParser => T): T

parse[T](command: String)(toResult: SqlBaseParser => T): T

parse sets up a proper ANTLR parsing infrastructure with SqlBaseLexer and SqlBaseParser (which are the ANTLR-specific classes of Spark SQL that are auto-generated at build time from the SqlBase.g4 grammar).

Tip	Review the definition of ANTLR grammar for Spark SQL in sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4.

Internally, parse first prints out the following INFO message to the logs:



INFO SparkSqlParser: Parsing command: [command]

INFO SparkSqlParser: Parsing command: [command]

Tip	Enable `INFO` logging level for the custom `AbstractSqlParser`, i.e. SparkSqlParser or CatalystSqlParser, to see the above INFO message.

parse then creates and sets up a SqlBaseLexer and SqlBaseParser that in turn passes the latter on to the input toResult function where the parsing finally happens.

Note	`parse` uses `SLL` prediction mode for parsing first before falling back to `LL` mode.

In case of parsing errors, parse reports a ParseException.

Note	`parse` is used in all the `parse` methods, i.e. parseDataType, parseExpression, parsePlan, parseTableIdentifier, and parseTableSchema.

`parseDataType` Method



parseDataType(sqlText: String): DataType

parseDataType(sqlText: String): DataType

Note	`parseDataType` is part of ParserInterface Contract to…FIXME.

parseDataType…FIXME

`parseExpression` Method



parseExpression(sqlText: String): Expression

parseExpression(sqlText: String): Expression

Note	`parseExpression` is part of ParserInterface Contract to…FIXME.

parseExpression…FIXME

`parseFunctionIdentifier` Method



parseFunctionIdentifier(sqlText: String): FunctionIdentifier

parseFunctionIdentifier(sqlText: String): FunctionIdentifier

Note	`parseFunctionIdentifier` is part of ParserInterface Contract to…FIXME.

parseFunctionIdentifier…FIXME

`parseTableIdentifier` Method



parseTableIdentifier(sqlText: String): TableIdentifier

parseTableIdentifier(sqlText: String): TableIdentifier

Note	`parseTableIdentifier` is part of ParserInterface Contract to…FIXME.

parseTableIdentifier…FIXME

`parseTableSchema` Method



parseTableSchema(sqlText: String): StructType

parseTableSchema(sqlText: String): StructType

Note	`parseTableSchema` is part of ParserInterface Contract to…FIXME.

parseTableSchema…FIXME

`parsePlan` Method



parsePlan(sqlText: String): LogicalPlan

parsePlan(sqlText: String): LogicalPlan

Note	`parsePlan` is part of ParserInterface Contract to…FIXME.

parsePlan creates a LogicalPlan for a given SQL textual statement.

Internally, parsePlan builds a SqlBaseParser and requests AstBuilder to parse a single SQL statement.

If a SQL statement could not be parsed, parsePlan throws a ParseException:



Unsupported SQL statement

Unsupported SQL statement

AbstractSqlParser — Base SQL Parsing Infrastructure