spark技术分享EliminateView Logical Optimization
EliminateView is a base logical optimization that removes (eliminates) View logical operators from a logical query plan.
EliminateView is part of the Finish Analysis once-executed batch in the standard batches of the Catalyst Optimizer.
EliminateView is simply a Catalyst rule for transforming logical plans, i.e. Rule[LogicalPlan].
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
val name = "demo_view" sql(s"CREATE OR REPLACE VIEW $name COMMENT 'demo view' AS VALUES 1,2") assert(spark.catalog.tableExists(name)) val q = spark.table(name) val analyzedPlan = q.queryExecution.analyzed scala> println(analyzedPlan.numberedTreeString) 00 SubqueryAlias demo_view 01 +- View (`default`.`demo_view`, [col1#37]) 02 +- Project [cast(col1#38 as int) AS col1#37] 03 +- LocalRelation [col1#38] import org.apache.spark.sql.catalyst.analysis.EliminateView val afterEliminateView = EliminateView(analyzedPlan) // Notice no View operator scala> println(afterEliminateView.numberedTreeString) 00 SubqueryAlias demo_view 01 +- Project [cast(col1#38 as int) AS col1#37] 02 +- LocalRelation [col1#38] // TIP: You may also want to use EliminateSubqueryAliases to eliminate SubqueryAliases |
Executing Rule — apply Method
|
1 2 3 4 5 |
apply(plan: LogicalPlan): LogicalPlan |
|
Note
|
apply is part of the Rule Contract to execute (apply) a rule on a TreeNode (e.g. LogicalPlan).
|
apply simply removes (eliminates) View unary logical operators from the input logical plan and replaces them with their child logical operator.
apply throws an AssertionError when the output schema of the View operator does not match the output schema of the child logical operator.
|
1 2 3 4 5 |
assertion failed: The output of the child [output] is different from the view output [output] |
|
Note
|
The assertion should not really happen since AliasViewChild logical analysis rule is executed earlier and takes care of not allowing for such difference in the output schema (by throwing an AnalysisException earlier).
|
