spark技术分享ExperimentalMethods
ExperimentalMethods holds extra optimizations and strategies that are used in SparkOptimizer and SparkPlanner, respectively.
| Name | Description | ||
|---|---|---|---|
|
|
Collection of rules to optimize LogicalPlans (i.e.
Used when |
||
|
|
Collection of SparkStrategies
Used when |
ExperimentalMethods is available as the experimental property of a SparkSession.
|
1 2 3 4 5 6 7 8 9 |
scala> :type spark org.apache.spark.sql.SparkSession scala> :type spark.experimental org.apache.spark.sql.ExperimentalMethods |
Example
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
import org.apache.spark.sql.catalyst.rules.Rule import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan object SampleRule extends Rule[LogicalPlan] { def apply(p: LogicalPlan): LogicalPlan = p } scala> :type spark org.apache.spark.sql.SparkSession spark.experimental.extraOptimizations = Seq(SampleRule) // extraOptimizations is used in Spark Optimizer val rule = spark.sessionState.optimizer.batches.flatMap(_.rules).filter(_ == SampleRule).head scala> rule.ruleName res0: String = SampleRule |
