spark技术分享UnresolvedAttribute Leaf Expression
UnresolvedAttribute is a named Attribute leaf expression (i.e. it has a name) that represents a reference to an entity in a logical query plan.
UnresolvedAttribute is created when:
-
AstBuilderis requested to visitDereference -
LogicalPlanis requested to resolve an attribute by name parts -
DescribeColumnCommandis executed
UnresolvedAttribute can never be resolved (and is replaced at analysis phase).
|
Note
|
|
Given UnresolvedAttribute can never be resolved it should not come as a surprise that it cannot be evaluated either (i.e. produce a value given an internal row). When requested to evaluate, UnresolvedAttribute simply reports a UnsupportedOperationException.
|
1 2 3 4 5 |
Cannot evaluate expression: [this] |
UnresolvedAttribute takes name parts when created.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute scala> val t1 = UnresolvedAttribute("t1") t1: org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute = 't1 scala> val t2 = UnresolvedAttribute("db1.t2") t2: org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute = 'db1.t2 scala> println(s"Number of name parts: ${t2.nameParts.length}") Number of name parts: 2 scala> println(s"Name parts: ${t2.nameParts.mkString(",")}") Name parts: db1,t2 |
UnresolvedAttribute can be created with a fully-qualified name with dots to separate name parts.
|
1 2 3 4 5 |
apply(name: String): UnresolvedAttribute |
|
Tip
|
Use backticks ( The following is a two-part attribute name with
|
UnresolvedAttribute can also be created without the dots with the special meaning.
|
1 2 3 4 5 6 7 8 |
import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute val attr1 = UnresolvedAttribute.quoted("a.b.c") scala> println(s"Number of name parts: ${attr1.nameParts.length}") Number of name parts: 1 |
|
Note
|
Catalyst DSL defines two Scala implicits to create an
Both implicits are part of ExpressionConversions Scala trait of Catalyst DSL. Import
|
|
Note
|
A
|
