spark技术分享ResolveCreateNamedStruct Logical Resolution Rule — Resolving NamePlaceholders In CreateNamedStruct Expressions
ResolveCreateNamedStruct is a logical resolution rule that replaces NamePlaceholders with Literals for the names in CreateNamedStruct expressions in an entire logical query plan.
ResolveCreateNamedStruct is part of the Resolution fixed-point batch in the standard batches of the Analyzer.
ResolveCreateNamedStruct is simply a Catalyst rule for transforming logical plans, i.e. Rule[LogicalPlan].
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
scala> :type spark org.apache.spark.sql.SparkSession val q = spark.range(1).select(struct($"id")) val logicalPlan = q.queryExecution.logical scala> println(logicalPlan.numberedTreeString) 00 'Project [unresolvedalias(named_struct(NamePlaceholder, 'id), None)] 01 +- AnalysisBarrier 02 +- Range (0, 1, step=1, splits=Some(8)) // Let's resolve references first import spark.sessionState.analyzer.ResolveReferences val planWithRefsResolved = ResolveReferences(logicalPlan) import org.apache.spark.sql.catalyst.analysis.ResolveCreateNamedStruct val afterResolveCreateNamedStruct = ResolveCreateNamedStruct(planWithRefsResolved) scala> println(afterResolveCreateNamedStruct.numberedTreeString) 00 'Project [unresolvedalias(named_struct(id, id#4L), None)] 01 +- AnalysisBarrier 02 +- Range (0, 1, step=1, splits=Some(8)) |
Executing Rule — apply Method
|
1 2 3 4 5 |
apply(plan: LogicalPlan): LogicalPlan |
|
Note
|
apply is part of the Rule Contract to execute (apply) a rule on a TreeNode (e.g. LogicalPlan).
|
apply traverses all Catalyst expressions (in the input LogicalPlan) that are CreateNamedStruct expressions which are not resolved yet and replaces NamePlaceholders with Literal expressions.
In other words, apply finds unresolved CreateNamedStruct expressions with NamePlaceholder expressions in the children and replaces them with the name of corresponding NamedExpression, but only if the NamedExpression is resolved.
In the end, apply creates a CreateNamedStruct with new children.
