spark技术分享UnresolvedGenerator Expression
UnresolvedGenerator is a Generator that represents an unresolved generator in a logical query plan.
UnresolvedGenerator is created exclusively when AstBuilder is requested to withGenerate (as part of Generate logical operator) for SQL’s LATERAL VIEW (in SELECT or FROM clauses).
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
// SQLs borrowed from https://cwiki.apache.org/confluence/display/Hive/LanguageManual+LateralView val sqlText = """ SELECT pageid, adid FROM pageAds LATERAL VIEW explode(adid_list) adTable AS adid """ // Register pageAds table Seq( ("front_page", Array(1, 2, 3)), ("contact_page", Array(3, 4, 5))) .toDF("pageid", "adid_list") .createOrReplaceTempView("pageAds") val query = sql(sqlText) val logicalPlan = query.queryExecution.logical scala> println(logicalPlan.numberedTreeString) 00 'Project ['pageid, 'adid] 01 +- 'Generate 'explode('adid_list), false, adtable, ['adid] 02 +- 'UnresolvedRelation `pageAds` import org.apache.spark.sql.catalyst.plans.logical.Generate val generator = logicalPlan.collectFirst { case g: Generate => g.generator }.get scala> :type generator org.apache.spark.sql.catalyst.expressions.Generator import org.apache.spark.sql.catalyst.analysis.UnresolvedGenerator scala> generator.isInstanceOf[UnresolvedGenerator] res1: Boolean = true |
UnresolvedGenerator can never be resolved (and is replaced at analysis phase).
Given UnresolvedGenerator can never be resolved it should not come as a surprise that it cannot be evaluated either (i.e. produce a value given an internal row). When requested to evaluate, UnresolvedGenerator simply reports a UnsupportedOperationException.
|
1 2 3 4 5 |
Cannot evaluate expression: [this] |
|
Note
|
|
|
Note
|
UnresolvedGenerator is similar to UnresolvedFunction and differs mostly by the type (to make Spark development with Scala easier?)
|
Creating UnresolvedGenerator Instance
UnresolvedGenerator takes the following when created:
-
Child expressions
