spark技术分享ConstantFolding Logical Optimization
ConstantFolding is a base logical optimization that replaces expressions that can be statically evaluated with their equivalent literal values.
ConstantFolding is part of the Operator Optimization before Inferring Filters fixed-point batch in the standard batches of the Catalyst Optimizer.
ConstantFolding is simply a Catalyst rule for transforming logical plans, i.e. Rule[LogicalPlan].
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
scala> spark.range(1).select(lit(3) > 2).explain(true) ... TRACE SparkOptimizer: === Applying Rule org.apache.spark.sql.catalyst.optimizer.ConstantFolding === !Project [(3 > 2) AS (3 > 2)#3] Project [true AS (3 > 2)#3] +- Range (0, 1, step=1, splits=Some(8)) +- Range (0, 1, step=1, splits=Some(8)) scala> spark.range(1).select('id + 'id > 0).explain(true) ... TRACE SparkOptimizer: === Applying Rule org.apache.spark.sql.catalyst.optimizer.ConstantFolding === !Project [((id#7L + id#7L) > cast(0 as bigint)) AS ((id + id) > 0)#10] Project [((id#7L + id#7L) > 0) AS ((id + id) > 0)#10] +- Range (0, 1, step=1, splits=Some(8)) +- Range (0, 1, step=1, splits=Some(8)) |
Executing Rule — apply Method
|
1 2 3 4 5 |
apply(plan: LogicalPlan): LogicalPlan |
|
Note
|
apply is part of the Rule Contract to execute (apply) a rule on a TreeNode (e.g. LogicalPlan).
|
apply…FIXME
