spark技术分享CreateTempViewUsing Logical Command
CreateTempViewUsing is a logical command for creating or replacing a temporary view (global or not) using a data source.
CreateTempViewUsing is created to represent CREATE TEMPORARY VIEW … USING SQL statements.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
val sqlText = s""" |CREATE GLOBAL TEMPORARY VIEW myTempCsvView |(id LONG, name STRING) |USING csv """.stripMargin // Logical commands are executed at analysis scala> sql(sqlText) res4: org.apache.spark.sql.DataFrame = [] scala> spark.catalog.listTables(spark.sharedState.globalTempViewManager.database).show +-------------+-----------+-----------+---------+-----------+ | name| database|description|tableType|isTemporary| +-------------+-----------+-----------+---------+-----------+ |mytempcsvview|global_temp| null|TEMPORARY| true| +-------------+-----------+-----------+---------+-----------+ |
Executing Logical Command — run Method
|
1 2 3 4 5 |
run(sparkSession: SparkSession): Seq[Row] |
|
Note
|
run is part of RunnableCommand Contract to execute (run) a logical command.
|
run creates a DataSource and requests it to resolve itself (i.e. create a BaseRelation).
run then requests the input SparkSession to create a DataFrame from the BaseRelation that is used to get the analyzed logical plan (that is the view definition of the temporary table).
Depending on the global flag, run requests the SessionCatalog to createGlobalTempView (global flag is on) or createTempView (global flag is off).
run throws an AnalysisException when executed with hive provider.
|
1 2 3 4 5 |
Hive data source can only be used with tables, you can't use it with CREATE TEMP VIEW USING |
Creating CreateTempViewUsing Instance
CreateTempViewUsing takes the following when created:
-
Optional user-defined schema (as StructType)
-
Name of the data source provider
argString Method
|
1 2 3 4 5 |
argString: String |
|
Note
|
argString is part of the TreeNode Contract to…FIXME.
|
argString…FIXME
