spark技术分享ResolveInlineTables Logical Resolution Rule
ResolveInlineTables is a logical resolution rule that resolves (replaces) UnresolvedInlineTable operators to LocalRelations in a logical query plan.
ResolveInlineTables is part of the Resolution fixed-point batch in the standard batches of the Analyzer.
ResolveInlineTables is simply a Catalyst rule for transforming logical plans, i.e. Rule[LogicalPlan].
ResolveInlineTables takes a SQLConf when created.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
val q = sql("VALUES 1, 2, 3") val plan = q.queryExecution.logical scala> println(plan.numberedTreeString) 00 'UnresolvedInlineTable [col1], [List(1), List(2), List(3)] scala> :type spark org.apache.spark.sql.SparkSession scala> :type spark.sessionState.conf org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.catalyst.analysis.ResolveInlineTables val rule = ResolveInlineTables(spark.sessionState.conf) val planAfterResolveInlineTables = rule(plan) scala> println(planAfterResolveInlineTables.numberedTreeString) 00 LocalRelation [col1#2] |
Executing Rule — apply Method
|
1 2 3 4 5 |
apply(plan: LogicalPlan): LogicalPlan |
|
Note
|
apply is part of the Rule Contract to execute (apply) a rule on a TreeNode (e.g. LogicalPlan).
|
apply simply searches the input plan up to find UnresolvedInlineTable logical operators with rows expressions resolved.
For such a UnresolvedInlineTable logical operator, apply validateInputDimension and validateInputEvaluable.
In the end, apply converts the UnresolvedInlineTable to a LocalRelation.
validateInputDimension Internal Method
|
1 2 3 4 5 |
validateInputDimension(table: UnresolvedInlineTable): Unit |
validateInputDimension…FIXME
|
Note
|
validateInputDimension is used exclusively when ResolveInlineTables logical resolution rule is executed.
|
validateInputEvaluable Internal Method
|
1 2 3 4 5 |
validateInputEvaluable(table: UnresolvedInlineTable): Unit |
validateInputEvaluable…FIXME
|
Note
|
validateInputEvaluable is used exclusively when ResolveInlineTables logical resolution rule is executed.
|
Converting UnresolvedInlineTable to LocalRelation — convert Internal Method
|
1 2 3 4 5 |
convert(table: UnresolvedInlineTable): LocalRelation |
convert…FIXME
|
Note
|
convert is used exclusively when ResolveInlineTables logical resolution rule is executed.
|
