UnresolvedAttribute is a named Attribute leaf expression (i.e. it has a name) that represents a reference to an entity in a logical query plan.
UnresolvedAttribute is created when:
Given UnresolvedAttribute can never be resolved it should not come as a surprise that it cannot be evaluated either (i.e. produce a value given an internal row). When requested to evaluate, UnresolvedAttribute simply reports a UnsupportedOperationException.
|
|
Cannot evaluate expression: [this] |
UnresolvedAttribute takes name parts when created.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute scala> val t1 = UnresolvedAttribute("t1") t1: org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute = 't1 scala> val t2 = UnresolvedAttribute("db1.t2") t2: org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute = 'db1.t2 scala> println(s"Number of name parts: ${t2.nameParts.length}") Number of name parts: 2 scala> println(s"Name parts: ${t2.nameParts.mkString(",")}") Name parts: db1,t2 |
UnresolvedAttribute can be created with a fully-qualified name with dots to separate name parts.
|
|
apply(name: String): UnresolvedAttribute |
|
Tip
|
Use backticks (.) to disable them as separators.
The following is a two-part attribute name with a.b and c name parts.
|
UnresolvedAttribute can also be created without the dots with the special meaning.
|
|
import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute val attr1 = UnresolvedAttribute.quoted("a.b.c") scala> println(s"Number of name parts: ${attr1.nameParts.length}") Number of name parts: 1 |
|
Note
|
Catalyst DSL defines two Scala implicits to create an UnresolvedAttribute:
-
StringToAttributeConversionHelper is a Scala implicit class that converts $"colName" into an UnresolvedAttribute
-
symbolToUnresolvedAttribute is a Scala implicit method that converts 'colName into an UnresolvedAttribute
Import expressions object to get access to the expression conversions.
|
|
// Use `sbt console` with Spark libraries defined (in `build.sbt`) import org.apache.spark.sql.catalyst.dsl.expressions._ scala> :type $"name" org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute val nameAttr: UnresolvedAttribute = 'name scala> :type nameAttr org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute |
|
|
Note
|
A UnresolvedAttribute can be replaced by (resolved) a NamedExpression using an analyzed logical plan (of the structured query the attribute is part of).
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
val analyzedPlan = Seq((0, "zero")).toDF("id", "name").queryExecution.analyzed import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute val nameAttr = UnresolvedAttribute("name") val nameResolved = analyzedPlan.resolveQuoted( name = nameAttr.name, resolver = spark.sessionState.analyzer.resolver).getOrElse(nameAttr) scala> println(nameResolved.numberedTreeString) 00 name#47: string scala> :type nameResolved org.apache.spark.sql.catalyst.expressions.NamedExpression |
|
spark技术分享
