spark技术分享UnresolvedFunction Unevaluable Expression — Logical Representation of Functions in Queries
UnresolvedFunction is an Catalyst expression that represents a function (application) in a logical query plan.
UnresolvedFunction is created as a result of the following:
-
callUDF standard function
-
RelationalGroupedDataset.agg operator with aggregation functions specified by name (that converts function names to UnresolvedFunction expressions)
-
AstBuilderis requested to visitFunctionCall (in SQL queries)
UnresolvedFunction can never be resolved (and is replaced at analysis phase).
|
Note
|
UnresolvedFunction is first looked up in LookupFunctions logical rule and then resolved in ResolveFunctions logical resolution rule.
|
Given UnresolvedFunction can never be resolved it should not come as a surprise that it cannot be evaluated either (i.e. produce a value given an internal row). When requested to evaluate, UnresolvedFunction simply reports a UnsupportedOperationException.
|
1 2 3 4 5 |
Cannot evaluate expression: [this] |
|
Note
|
Unevaluable expressions are expressions that have to be replaced by some other expressions during analysis or optimization (or they fail analysis). |
|
Tip
|
Use Catalyst DSL’s function or distinctFunction to create a UnresolvedFunction with isDistinct flag off and on, respectively.
|
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
// Using Catalyst DSL to create UnresolvedFunctions import org.apache.spark.sql.catalyst.dsl.expressions._ // Scala Symbols supported only val f = 'f.function() scala> :type f org.apache.spark.sql.catalyst.analysis.UnresolvedFunction scala> f.isDistinct res0: Boolean = false val g = 'g.distinctFunction() scala> g.isDistinct res1: Boolean = true |
Creating UnresolvedFunction (With Database Undefined) — apply Factory Method
|
1 2 3 4 5 |
apply(name: String, children: Seq[Expression], isDistinct: Boolean): UnresolvedFunction |
apply creates a FunctionIdentifier with the name and no database first and then creates a UnresolvedFunction with the FunctionIdentifier, children and isDistinct flag.
|
Note
|
|
Creating UnresolvedFunction Instance
UnresolvedFunction takes the following when created:
-
Child expressions
