spark技术分享PhysicalOperation — Scala Extractor for Destructuring Logical Query Plans
PhysicalOperation is a Scala extractor to destructure a logical query plan into a tuple with the following elements:
-
Named expressions (aka projects)
-
Expressions (aka filters)
-
Logical operator (aka leaf node)
|
1 2 3 4 5 6 |
ReturnType (Seq[NamedExpression], Seq[Expression], LogicalPlan) |
The following idiom is often used in Strategy implementations (e.g. HiveTableScans, InMemoryScans, DataSourceStrategy, FileSourceStrategy):
|
1 2 3 4 5 6 7 8 9 |
def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match { case PhysicalOperation(projections, predicates, plan) => // do something case _ => Nil } |
Whenever used to pattern match to a LogicalPlan, PhysicalOperation‘s unapply is called.
unapply Method
|
1 2 3 4 5 6 7 |
type ReturnType = (Seq[NamedExpression], Seq[Expression], LogicalPlan) unapply(plan: LogicalPlan): Option[ReturnType] |
unapply…FIXME
|
Note
|
unapply is almost collectProjectsAndFilters method itself (with some manipulations of the return value).
|
|
Note
|
|
