spark技术分享View Unary Logical Operator
View is a logical operator with a single child logical operator.
View is created exclusively when SessionCatalog is requested to find a relation in the catalogs (e.g. when DescribeTableCommand logical command is executed and the table type is VIEW).
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
// Let's create a view first // Using SQL directly to manage views is so much nicer val name = "demo_view" sql(s"CREATE OR REPLACE VIEW $name COMMENT 'demo view' AS VALUES 1,2") assert(spark.catalog.tableExists(name)) val q = sql(s"DESC EXTENDED $name") val allRowsIncluded = 100 scala> q.show(numRows = allRowsIncluded) +--------------------+--------------------+-------+ | col_name| data_type|comment| +--------------------+--------------------+-------+ | col1| int| null| | | | | |# Detailed Table ...| | | | Database| default| | | Table| demo_view| | | Owner| jacek| | | Created Time|Thu Aug 30 08:55:...| | | Last Access|Thu Jan 01 01:00:...| | | Created By| Spark 2.3.1| | | Type| VIEW| | | Comment| demo view| | | View Text| VALUES 1,2| | |View Default Data...| default| | |View Query Output...| [col1]| | | Table Properties|[transient_lastDd...| | | Serde Library|org.apache.hadoop...| | | InputFormat|org.apache.hadoop...| | | OutputFormat|org.apache.hadoop...| | | Storage Properties|[serialization.fo...| | +--------------------+--------------------+-------+ |
View is a MultiInstanceRelation so a new instance will be created to appear multiple times in a physical query plan. When requested for a new instance, View creates new instances of the output attributes.
View has the following simple description (with state prefix):
|
1 2 3 4 5 |
View ([identifier], [output]) |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
val name = "demo_view" sql(s"CREATE OR REPLACE VIEW $name COMMENT 'demo view' AS VALUES 1,2") assert(spark.catalog.tableExists(name)) val q = spark.table(name) val qe = q.queryExecution val logicalPlan = qe.logical scala> println(logicalPlan.simpleString) 'UnresolvedRelation `demo_view` val analyzedPlan = qe.analyzed scala> println(analyzedPlan.numberedTreeString) 00 SubqueryAlias demo_view 01 +- View (`default`.`demo_view`, [col1#33]) 02 +- Project [cast(col1#34 as int) AS col1#33] 03 +- LocalRelation [col1#34] // Skip SubqueryAlias scala> println(analyzedPlan.children.head.simpleString) View (`default`.`demo_view`, [col1#33]) |
|
Note
|
View is resolved by ResolveRelations logical resolution.
|
|
Note
|
AliasViewChild logical analysis rule makes sure that the output of a View matches the output of the child logical operator.
|
|
Note
|
EliminateView logical optimization removes (eliminates) View operators from a logical query plan.
|
Creating View Instance
View takes the following when created:
-
Output schema attributes (as
Seq[Attribute]) -
Child logical plan
