spark技术分享SparkSessionExtensions
SparkSessionExtensions is an interface that a Spark developer can use to extend a SparkSession with custom query execution rules and a relational entity parser.
As a Spark developer, you use Builder.withExtensions method (while building a new SparkSession) to access the session-bound SparkSessionExtensions.
| Method | Description | ||
|---|---|---|---|
|
|||
|
Registering a custom operator optimization rule
|
|||
|
|||
|
|||
|
|||
|
SparkSessionExtensions is an integral part of SparkSession (and is indirectly required to create one).
| Name | Description |
|---|---|
|
|
Used when |
Associating Custom Operator Optimization Rules with SparkSession — buildOptimizerRules Method
|
1 2 3 4 5 |
buildOptimizerRules(session: SparkSession): Seq[Rule[LogicalPlan]] |
buildOptimizerRules gives the optimizerRules logical rules that are associated with the input SparkSession.
|
Note
|
buildOptimizerRules is used exclusively when BaseSessionStateBuilder is requested for the custom operator optimization rules to add to the base Operator Optimization batch.
|
Registering Custom Check Analysis Rule (Builder) — injectCheckRule Method
|
1 2 3 4 5 |
injectCheckRule(builder: SparkSession => LogicalPlan => Unit): Unit |
injectCheckRule…FIXME
Registering Custom Operator Optimization Rule (Builder) — injectOptimizerRule Method
|
1 2 3 4 5 |
injectOptimizerRule(builder: SparkSession => Rule[LogicalPlan]): Unit |
injectOptimizerRule simply registers a custom operator optimization rule (as a RuleBuilder function) to the optimizerRules internal registry.
Registering Custom Parser (Builder) — injectParser Method
|
1 2 3 4 5 |
injectParser(builder: (SparkSession, ParserInterface) => ParserInterface): Unit |
injectParser…FIXME
Registering Custom Planner Strategy (Builder) — injectPlannerStrategy Method
|
1 2 3 4 5 |
injectPlannerStrategy(builder: SparkSession => Strategy): Unit |
injectPlannerStrategy…FIXME
