spark技术分享Subexpression Elimination In Code-Generated Expression Evaluation (Common Expression Reuse)
Subexpression Elimination (aka Common Expression Reuse) is an optimisation of a logical query plan that eliminates expressions in code-generated (non-interpreted) expression evaluation.
Subexpression Elimination is enabled by default. Use the internal spark.sql.subexpressionElimination.enabled configuration property control whether the feature is enabled (true) or not (false).
Subexpression Elimination is used (by means of subexpressionEliminationEnabled flag of SparkPlan) when the following physical operators are requested to execute (i.e. moving away from queries to an RDD of internal rows to describe a distributed computation):
Internally, subexpression elimination happens when CodegenContext is requested for subexpressionElimination (when CodegenContext is requested to generateExpressions with subexpression elimination enabled).
spark.sql.subexpressionElimination.enabled Configuration Property
spark.sql.subexpressionElimination.enabled internal configuration property controls whether the subexpression elimination optimization is enabled or not.
|
Tip
|
Use subexpressionEliminationEnabled method to access the current value. |
|
1 2 3 4 5 6 7 8 9 |
scala> import spark.sessionState.conf import spark.sessionState.conf scala> conf.subexpressionEliminationEnabled res1: Boolean = true |
