spark技术分享UnresolvedOrdinal Unevaluable Leaf Expression
UnresolvedOrdinal is a leaf expression that represents a single integer literal in Sort logical operators (in SortOrder ordering expressions) and in Aggregate logical operators (in grouping expressions) in a logical plan.
UnresolvedOrdinal is created when SubstituteUnresolvedOrdinals logical resolution rule is executed.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
// Note "order by 1" clause val sqlText = "select id from VALUES 1, 2, 3 t1(id) order by 1" val logicalPlan = spark.sql(sqlText).queryExecution.logical scala> println(logicalPlan.numberedTreeString) 00 'Sort [1 ASC NULLS FIRST], true 01 +- 'Project ['id] 02 +- 'SubqueryAlias t1 03 +- 'UnresolvedInlineTable [id], [List(1), List(2), List(3)] import org.apache.spark.sql.catalyst.analysis.SubstituteUnresolvedOrdinals val rule = new SubstituteUnresolvedOrdinals(spark.sessionState.conf) val logicalPlanWithUnresolvedOrdinals = rule.apply(logicalPlan) scala> println(logicalPlanWithUnresolvedOrdinals.numberedTreeString) 00 'Sort [unresolvedordinal(1) ASC NULLS FIRST], true 01 +- 'Project ['id] 02 +- 'SubqueryAlias t1 03 +- 'UnresolvedInlineTable [id], [List(1), List(2), List(3)] import org.apache.spark.sql.catalyst.plans.logical.Sort val sortOp = logicalPlanWithUnresolvedOrdinals.collect { case s: Sort => s }.head val sortOrder = sortOp.order.head import org.apache.spark.sql.catalyst.analysis.UnresolvedOrdinal val unresolvedOrdinalExpr = sortOrder.child.asInstanceOf[UnresolvedOrdinal] scala> println(unresolvedOrdinalExpr) unresolvedordinal(1) |
UnresolvedOrdinal takes a single ordinal integer when created.
UnresolvedOrdinal is an unevaluable expression and cannot be evaluated (i.e. produce a value given an internal row).
|
Note
|
An unevaluable expression cannot be evaluated to produce a value (neither in interpreted nor code-generated expression evaluations) and has to be resolved (replaced) to some other expressions or logical operators at analysis or optimization phases or they fail analysis. |
UnresolvedOrdinal can never be resolved (and is replaced at analysis phase).
|
Note
|
UnresolvedOrdinal is resolved when ResolveOrdinalInOrderByAndGroupBy logical resolution rule is executed.
|
UnresolvedOrdinal has no representation in SQL.
|
Note
|
UnresolvedOrdinal in GROUP BY ordinal position is not allowed for a select list with a star (*).
|
