spark技术分享CombineTypedFilters Logical Optimization
CombineTypedFilters is a base logical optimization that combines two back to back (typed) filters into one that ultimately ends up as a single method call.
CombineTypedFilters is part of the Object Expressions Optimization fixed-point batch in the standard batches of the Catalyst Optimizer.
CombineTypedFilters is simply a Catalyst rule for transforming logical plans, i.e. Rule[LogicalPlan].
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
scala> :type spark org.apache.spark.sql.SparkSession // A query with two consecutive typed filters val q = spark.range(10).filter(_ % 2 == 0).filter(_ == 0) scala> q.queryExecution.optimizedPlan ... TRACE SparkOptimizer: === Applying Rule org.apache.spark.sql.catalyst.optimizer.CombineTypedFilters === TypedFilter <function1>, class java.lang.Long, [StructField(value,LongType,true)], newInstance(class java.lang.Long) TypedFilter <function1>, class java.lang.Long, [StructField(value,LongType,true)], newInstance(class java.lang.Long) !+- TypedFilter <function1>, class java.lang.Long, [StructField(value,LongType,true)], newInstance(class java.lang.Long) +- Range (0, 10, step=1, splits=Some(8)) ! +- Range (0, 10, step=1, splits=Some(8)) TRACE SparkOptimizer: Fixed point reached for batch Typed Filter Optimization after 2 iterations. DEBUG SparkOptimizer: === Result of Batch Typed Filter Optimization === TypedFilter <function1>, class java.lang.Long, [StructField(value,LongType,true)], newInstance(class java.lang.Long) TypedFilter <function1>, class java.lang.Long, [StructField(value,LongType,true)], newInstance(class java.lang.Long) !+- TypedFilter <function1>, class java.lang.Long, [StructField(value,LongType,true)], newInstance(class java.lang.Long) +- Range (0, 10, step=1, splits=Some(8)) ! +- Range (0, 10, step=1, splits=Some(8)) ... |
Executing Rule — apply Method
|
1 2 3 4 5 |
apply(plan: LogicalPlan): LogicalPlan |
|
Note
|
apply is part of the Rule Contract to execute (apply) a rule on a TreeNode (e.g. LogicalPlan).
|
apply…FIXME
