spark技术分享UnresolvedCatalogRelation Leaf Logical Operator — Placeholder of Catalog Tables
UnresolvedCatalogRelation is a leaf logical operator that acts as a placeholder in a logical query plan until FindDataSourceTable logical evaluation rule resolves it to a concrete relation logical plan (i.e. a LogicalRelation for a data source table or a HiveTableRelation for hive table).
UnresolvedCatalogRelation is created when SessionCatalog is requested to find a relation (for DescribeTableCommand logical command or ResolveRelations logical evaluation rule).
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
// non-temporary global or local view // database defined // externalCatalog.getTable returns non-VIEW table // Make the example reproducible val tableName = "t1" spark.sharedState.externalCatalog.dropTable( db = "default", table = tableName, ignoreIfNotExists = true, purge = true) spark.range(10).write.saveAsTable(tableName) scala> :type spark.sessionState.catalog org.apache.spark.sql.catalyst.catalog.SessionCatalog import org.apache.spark.sql.catalyst.TableIdentifier val plan = spark.sessionState.catalog.lookupRelation(TableIdentifier(tableName)) scala> println(plan.numberedTreeString) 00 'SubqueryAlias t1 01 +- 'UnresolvedCatalogRelation `default`.`t1`, org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe |
When created, UnresolvedCatalogRelation asserts that the database is specified.
UnresolvedCatalogRelation can never be resolved and is converted to a LogicalRelation for a data source table or a HiveTableRelation for hive table at analysis phase.
UnresolvedCatalogRelation uses an empty output schema.
UnresolvedCatalogRelation takes a single CatalogTable when created.
