spark技术分享CatalystSqlParser — DataTypes and StructTypes Parser
CatalystSqlParser is a AbstractSqlParser with AstBuilder as the required astBuilder.
CatalystSqlParser is used to translate DataTypes from their canonical string representation (e.g. when adding fields to a schema or casting column to a different data type) or StructTypes.
|
1 2 3 4 5 6 7 8 9 10 |
import org.apache.spark.sql.types.StructType scala> val struct = new StructType().add("a", "int") struct: org.apache.spark.sql.types.StructType = StructType(StructField(a,IntegerType,true)) scala> val asInt = expr("token = 'hello'").cast("int") asInt: org.apache.spark.sql.Column = CAST((token = hello) AS INT) |
When parsing, you should see INFO messages in the logs:
|
1 2 3 4 5 |
INFO CatalystSqlParser: Parsing command: int |
It is also used in HiveClientImpl (when converting columns from Hive to Spark) and in OrcFileOperator (when inferring the schema for ORC files).
|
Tip
|
Enable Add the following line to
Refer to Logging. |
