QueryPlanner — Converting Logical Plan to Physical Trees-spark技术分享

QueryPlanner — Converting Logical Plan to Physical Trees

QueryPlanner plans a logical plan for execution, i.e. converts a logical plan to one or more physical plans using strategies.

Note	`QueryPlanner` generates at least one physical plan.

QueryPlanner‘s main method is plan that defines the extension points, i.e. strategies, collectPlaceholders and prunePlans.

QueryPlanner is part of Catalyst Framework.

QueryPlanner Contract



abstract class QueryPlanner[PhysicalPlan <: TreeNode[PhysicalPlan]] {
  def collectPlaceholders(plan: PhysicalPlan): Seq[(PhysicalPlan, LogicalPlan)]
  def prunePlans(plans: Iterator[PhysicalPlan]): Iterator[PhysicalPlan]
  def strategies: Seq[GenericStrategy[PhysicalPlan]]
}

abstract class QueryPlanner[PhysicalPlan <: TreeNode[PhysicalPlan]] {

def collectPlaceholders(plan: PhysicalPlan): Seq[(PhysicalPlan, LogicalPlan)]

def prunePlans(plans: Iterator[PhysicalPlan]): Iterator[PhysicalPlan]

def strategies: Seq[GenericStrategy[PhysicalPlan]]

}

Table 1. QueryPlanner Contract
Method	Description
`strategies`	Collection of GenericStrategy planning strategies. Used exclusively as an extension point in plan.
`collectPlaceholders`	Collection of “placeholder” physical plans and the corresponding logical plans. Used exclusively as an extension point in plan. Overriden in SparkPlanner
`prunePlans`	Prunes physical plans (e.g. bad or somehow incorrect plans). Used exclusively as an extension point in plan.

Planning Logical Plan — `plan` Method



plan(plan: LogicalPlan): Iterator[PhysicalPlan]

plan(plan: LogicalPlan): Iterator[PhysicalPlan]

plan converts the input plan logical plan to zero or more PhysicalPlan plans.

Internally, plan applies planning strategies to the input plan (one by one collecting all as the plan candidates).

plan then walks over the plan candidates to collect placeholders.

If a plan does not contain a placeholder, the plan is returned as is. Otherwise, plan walks over placeholders (as pairs of PhysicalPlan and unplanned logical plan) and (recursively) plans the child logical plan. plan then replaces the placeholders with the planned child logical plan.

In the end, plan prunes “bad” physical plans.

Note	`plan` is used exclusively (through the concrete SparkPlanner) when a `QueryExecution` is requested for a physical plan.

QueryPlanner — Converting Logical Plan to Physical Trees