spark技术分享LookupFunctions Logical Rule — Checking Whether UnresolvedFunctions Are Resolvable
LookupFunctions is a logical rule that the logical query plan analyzer uses to make sure that UnresolvedFunction expressions can be resolved in an entire logical query plan.
LookupFunctions is similar to ResolveFunctions logical resolution rule, but it is ResolveFunctions to resolve UnresolvedFunction expressions while LookupFunctions is just a sanity check that a future resolution is possible if tried.
Technically, LookupFunctions is just a Catalyst rule for transforming logical plans, i.e. Rule[LogicalPlan].
|
Note
|
LookupFunctions does not however transform a logical plan.
|
LookupFunctions is part of Simple Sanity Check one-off batch of rules.
|
Note
|
LookupFunctions is a Scala object inside Analyzer class.
|
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
// Using Catalyst DSL to create a logical plan with an unregistered function import org.apache.spark.sql.catalyst.dsl.plans._ val t1 = table("t1") import org.apache.spark.sql.catalyst.dsl.expressions._ val f1 = 'f1.function() val plan = t1.select(f1) scala> println(plan.numberedTreeString) 00 'Project [unresolvedalias('f1(), None)] 01 +- 'UnresolvedRelation `t1` // Make sure the function f1 does not exist import org.apache.spark.sql.catalyst.FunctionIdentifier spark.sessionState.catalog.dropFunction(FunctionIdentifier("f1"), ignoreIfNotExists = true) assert(spark.catalog.functionExists("f1") == false) import spark.sessionState.analyzer.LookupFunctions scala> LookupFunctions(plan) org.apache.spark.sql.AnalysisException: Undefined function: 'f1'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'.; at org.apache.spark.sql.catalyst.analysis.Analyzer$LookupFunctions$$anonfun$apply$15$$anonfun$applyOrElse$49.apply(Analyzer.scala:1198) at org.apache.spark.sql.catalyst.analysis.Analyzer$LookupFunctions$$anonfun$apply$15$$anonfun$applyOrElse$49.apply(Analyzer.scala:1198) at org.apache.spark.sql.catalyst.analysis.package$.withPosition(package.scala:48) at org.apache.spark.sql.catalyst.analysis.Analyzer$LookupFunctions$$anonfun$apply$15.applyOrElse(Analyzer.scala:1197) at org.apache.spark.sql.catalyst.analysis.Analyzer$LookupFunctions$$anonfun$apply$15.applyOrElse(Analyzer.scala:1195) |
Applying LookupFunctions to Logical Plan — apply Method
|
1 2 3 4 5 |
apply(plan: LogicalPlan): LogicalPlan |
|
Note
|
apply is part of Rule Contract to apply a rule to a logical plan.
|
apply finds all UnresolvedFunction expressions (in every logical operator in the input logical plan) and requests the SessionCatalog to check if their functions exist.
apply does nothing if a function exists or reports a NoSuchFunctionException (that fails logical analysis).
|
1 2 3 4 5 |
Undefined function: '[func]'. This function is neither a registered temporary function nor a permanent function registered in the database '[db]'. |
