spark技术分享UnresolvedHint Unary Logical Operator — Attaching Hint to Logical Plan
UnresolvedHint is a unary logical operator that represents a hint (by name and parameters) for the child logical plan.
UnresolvedHint is created and added to a logical plan when:
-
Dataset.hint operator is used
-
AstBuilderconverts/*+ hint */inSELECTSQL queries
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
// Dataset API val q = spark.range(1).hint("myHint", 100, true) val plan = q.queryExecution.logical scala> println(plan.numberedTreeString) 00 'UnresolvedHint myHint, [100, true] 01 +- Range (0, 1, step=1, splits=Some(8)) // SQL val q = sql("SELECT /*+ myHint (100, true) */ 1") val plan = q.queryExecution.logical scala> println(plan.numberedTreeString) 00 'UnresolvedHint myHint, [100, true] 01 +- 'Project [unresolvedalias(1, None)] 02 +- OneRowRelation |
When created UnresolvedHint takes:
-
Child logical plan
UnresolvedHint can never be resolved and is supposed to be converted to a ResolvedHint unary logical operator during query analysis (or simply removed from a logical plan).
|
Note
|
There are the following logical rules that Spark Analyzer uses to analyze logical plans with the UnresolvedHint logical operator:
The order of executing the above rules matters. |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
// Let's hint the query twice // The order of hints matters as every hint operator executes Spark analyzer // That will resolve all but the last hint val q = spark.range(100). hint("broadcast"). hint("myHint", 100, true) val plan = q.queryExecution.logical scala> println(plan.numberedTreeString) 00 'UnresolvedHint myHint, [100, true] 01 +- ResolvedHint (broadcast) 02 +- Range (0, 100, step=1, splits=Some(8)) // Let's resolve unresolved hints import org.apache.spark.sql.catalyst.rules.RuleExecutor import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan import org.apache.spark.sql.catalyst.analysis.ResolveHints import org.apache.spark.sql.internal.SQLConf object HintResolver extends RuleExecutor[LogicalPlan] { lazy val batches = Batch("Hints", FixedPoint(maxIterations = 100), new ResolveHints.ResolveBroadcastHints(SQLConf.get), ResolveHints.RemoveAllHints) :: Nil } val resolvedPlan = HintResolver.execute(plan) scala> println(resolvedPlan.numberedTreeString) 00 ResolvedHint (broadcast) 01 +- Range (0, 100, step=1, splits=Some(8)) |
UnresolvedHint uses the child operator’s output schema for yours.
|
Tip
|
Use
|
