spark技术分享InMemoryScans Execution Planning Strategy
InMemoryScans is an execution planning strategy that plans InMemoryRelation logical operators to InMemoryTableScanExec physical operators.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
val spark: SparkSession = ... // query uses InMemoryRelation logical operator val q = spark.range(5).cache val plan = q.queryExecution.optimizedPlan scala> println(plan.numberedTreeString) 00 InMemoryRelation [id#208L], true, 10000, StorageLevel(disk, memory, deserialized, 1 replicas) 01 +- *Range (0, 5, step=1, splits=8) // InMemoryScans is an internal class of SparkStrategies import spark.sessionState.planner.InMemoryScans val physicalPlan = InMemoryScans.apply(plan).head scala> println(physicalPlan.numberedTreeString) 00 InMemoryTableScan [id#208L] 01 +- InMemoryRelation [id#208L], true, 10000, StorageLevel(disk, memory, deserialized, 1 replicas) 02 +- *Range (0, 5, step=1, splits=8) |
InMemoryScans is part of the standard execution planning strategies of SparkPlanner.
Applying InMemoryScans Strategy to Logical Plan (Executing InMemoryScans) — apply Method
|
1 2 3 4 5 |
apply(plan: LogicalPlan): Seq[SparkPlan] |
|
Note
|
apply is part of GenericStrategy Contract to generate a collection of SparkPlans for a given logical plan.
|
apply requests PhysicalOperation extractor to destructure the input logical plan to a InMemoryRelation logical operator.
In the end, apply pruneFilterProject with a new InMemoryTableScanExec physical operator.
