spark技术分享ResolveCoalesceHints Logical Resolution Rule — Resolving UnresolvedHint Operators with COALESCE and REPARTITION Hints
ResolveCoalesceHints is a logical resolution rule that the Spark Analyzer uses to resolve UnresolvedHint logical operators with COALESCE or REPARTITION hints (case-insensitive) to ResolvedHint operators.
COALESCE or REPARTITION hints expect a partition number as the only parameter.
Technically, ResolveCoalesceHints is a Catalyst rule for transforming logical plans, i.e. Rule[LogicalPlan].
ResolveCoalesceHints is part of Hints fixed-point batch of rules (that is executed before any other rule).
ResolveCoalesceHints takes no input parameters when created.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
Example: Using COALESCE Hint // Use Catalyst DSL to create a logical plan import org.apache.spark.sql.catalyst.dsl.plans._ val plan = table("t1").hint(name = "COALESCE", 3) scala> println(plan.numberedTreeString) 00 'UnresolvedHint COALESCE, [3] 01 +- 'UnresolvedRelation `t1` import org.apache.spark.sql.catalyst.analysis.ResolveHints.ResolveCoalesceHints val analyzedPlan = ResolveCoalesceHints(plan) scala> println(analyzedPlan.numberedTreeString) 00 'Repartition 3, false 01 +- 'UnresolvedRelation `t1` |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
Example: Using REPARTITION Hint // Use Catalyst DSL to create a logical plan import org.apache.spark.sql.catalyst.dsl.plans._ val plan = table("t1").hint(name = "REPARTITION", 3) scala> println(plan.numberedTreeString) 00 'UnresolvedHint REPARTITION, [3] 01 +- 'UnresolvedRelation `t1` import org.apache.spark.sql.catalyst.analysis.ResolveHints.ResolveCoalesceHints val analyzedPlan = ResolveCoalesceHints(plan) scala> println(analyzedPlan.numberedTreeString) 00 'Repartition 3, true 01 +- 'UnresolvedRelation `t1` |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
Example: Using COALESCE Hint in SQL val q = sql("SELECT /*+ COALESCE(10) */ * FROM VALUES 1 t(id)") val plan = q.queryExecution.logical scala> println(plan.numberedTreeString) 00 'UnresolvedHint COALESCE, [10] 01 +- 'Project [*] 02 +- 'SubqueryAlias `t` 03 +- 'UnresolvedInlineTable [id], [List(1)] import org.apache.spark.sql.catalyst.analysis.ResolveHints.ResolveCoalesceHints val analyzedPlan = ResolveCoalesceHints(plan) scala> println(analyzedPlan.numberedTreeString) 00 'Repartition 10, false 01 +- 'Project [*] 02 +- 'SubqueryAlias `t` 03 +- 'UnresolvedInlineTable [id], [List(1)] |
