View-spark技术分享

View Unary Logical Operator

View is a logical operator with a single child logical operator.

View is created exclusively when SessionCatalog is requested to find a relation in the catalogs (e.g. when DescribeTableCommand logical command is executed and the table type is VIEW).


// Let's create a view first
// Using SQL directly to manage views is so much nicer
val name = "demo_view"
sql(s"CREATE OR REPLACE VIEW $name COMMENT 'demo view' AS VALUES 1,2")
assert(spark.catalog.tableExists(name))

val q = sql(s"DESC EXTENDED $name")

val allRowsIncluded = 100
scala> q.show(numRows = allRowsIncluded)
+--------------------+--------------------+-------+
|            col_name|           data_type|comment|
+--------------------+--------------------+-------+
|                col1|                 int|   null|
|                    |                    |       |
|# Detailed Table ...|                    |       |
|            Database|             default|       |
|               Table|           demo_view|       |
|               Owner|               jacek|       |
|        Created Time|Thu Aug 30 08:55:...|       |
|         Last Access|Thu Jan 01 01:00:...|       |
|          Created By|         Spark 2.3.1|       |
|                Type|                VIEW|       |
|             Comment|           demo view|       |
|           View Text|          VALUES 1,2|       |
|View Default Data...|             default|       |
|View Query Output...|              [col1]|       |
|    Table Properties|[transient_lastDd...|       |
|       Serde Library|org.apache.hadoop...|       |
|         InputFormat|org.apache.hadoop...|       |
|        OutputFormat|org.apache.hadoop...|       |
|  Storage Properties|[serialization.fo...|       |
+--------------------+--------------------+-------+

// Let's create a view first

// Using SQL directly to manage views is so much nicer

val name = "demo_view"

sql(s"CREATE OR REPLACE VIEW $name COMMENT 'demo view' AS VALUES 1,2")

assert(spark.catalog.tableExists(name))

val q = sql(s"DESC EXTENDED $name")

val allRowsIncluded = 100

scala> q.show(numRows = allRowsIncluded)

+--------------------+--------------------+-------+

| col_name| data_type|comment|

+--------------------+--------------------+-------+

| col1| int| null|

| | | |

|# Detailed Table ...| | |

| Database| default| |

| Table| demo_view| |

| Owner| jacek| |

| Created Time|Thu Aug 30 08:55:...| |

| Last Access|Thu Jan 01 01:00:...| |

| Created By| Spark 2.3.1| |

| Type| VIEW| |

| Comment| demo view| |

| View Text| VALUES 1,2| |

|View Default Data...| default| |

|View Query Output...| [col1]| |

| Table Properties|[transient_lastDd...| |

| Serde Library|org.apache.hadoop...| |

| InputFormat|org.apache.hadoop...| |

| OutputFormat|org.apache.hadoop...| |

| Storage Properties|[serialization.fo...| |

+--------------------+--------------------+-------+

View is a MultiInstanceRelation so a new instance will be created to appear multiple times in a physical query plan. When requested for a new instance, View creates new instances of the output attributes.

View is considered resolved only when the child is.

View has the following simple description (with state prefix):


View ([identifier], [output])

View ([identifier], [output])


val name = "demo_view"
sql(s"CREATE OR REPLACE VIEW $name COMMENT 'demo view' AS VALUES 1,2")
assert(spark.catalog.tableExists(name))

val q = spark.table(name)
val qe = q.queryExecution

val logicalPlan = qe.logical
scala> println(logicalPlan.simpleString)
'UnresolvedRelation `demo_view`

val analyzedPlan = qe.analyzed
scala> println(analyzedPlan.numberedTreeString)
00 SubqueryAlias demo_view
01 +- View (`default`.`demo_view`, [col1#33])
02    +- Project [cast(col1#34 as int) AS col1#33]
03       +- LocalRelation [col1#34]

// Skip SubqueryAlias
scala> println(analyzedPlan.children.head.simpleString)
View (`default`.`demo_view`, [col1#33])

val name = "demo_view"

sql(s"CREATE OR REPLACE VIEW $name COMMENT 'demo view' AS VALUES 1,2")

assert(spark.catalog.tableExists(name))

val q = spark.table(name)

val qe = q.queryExecution

val logicalPlan = qe.logical

scala> println(logicalPlan.simpleString)

'UnresolvedRelation `demo_view`

val analyzedPlan = qe.analyzed

scala> println(analyzedPlan.numberedTreeString)

00 SubqueryAlias demo_view

01 +- View (`default`.`demo_view`, [col1#33])

02 +- Project [cast(col1#34 as int) AS col1#33]

03 +- LocalRelation [col1#34]

// Skip SubqueryAlias

scala> println(analyzedPlan.children.head.simpleString)

View (`default`.`demo_view`, [col1#33])

Note	`View` is resolved by ResolveRelations logical resolution.

Note	AliasViewChild logical analysis rule makes sure that the output of a `View` matches the output of the child logical operator.

Note	EliminateView logical optimization removes (eliminates) `View` operators from a logical query plan.

Note	Inserting into a view is not allowed.

Creating View Instance

View takes the following when created:

CatalogTable
Output schema attributes (as Seq[Attribute])
Child logical plan

View

View Unary Logical Operator

Creating View Instance

相关推荐

欢迎关注：spark技术分享

热门标签

近期文章

分类目录

关注公众号：spark技术分享

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏

QQ咨询

回顶部

View Unary Logical Operator

Creating View Instance

相关推荐

欢迎关注：spark技术分享

热门标签

近期文章

分类目录

关注公众号：spark技术分享

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏

QQ咨询

回顶部

切换注册登录

用户名或邮箱

密码

切换登录注册

昵称

邮箱