BaseSessionStateBuilder — Generic Builder of SessionState-spark技术分享

BaseSessionStateBuilder — Generic Builder of SessionState

BaseSessionStateBuilder is the contract of builder objects that coordinate construction of a new SessionState.

Table 1. BaseSessionStateBuilders
BaseSessionStateBuilder	Description
SessionStateBuilder
HiveSessionStateBuilder

BaseSessionStateBuilder is created when SparkSession is requested for a SessionState.



scala> :type spark
org.apache.spark.sql.SparkSession

scala> :type spark.sessionState
org.apache.spark.sql.internal.SessionState

scala> :type spark

org.apache.spark.sql.SparkSession

scala> :type spark.sessionState

org.apache.spark.sql.internal.SessionState

BaseSessionStateBuilder requires that implementations define newBuilder method that SparkSession uses (indirectly) when requested for the SessionState (per spark.sql.catalogImplementation internal configuration property).



newBuilder: (SparkSession, Option[SessionState]) => BaseSessionStateBuilder

newBuilder: (SparkSession, Option[SessionState]) => BaseSessionStateBuilder

Note	`BaseSessionStateBuilder` and spark.sql.catalogImplementation configuration property allow for Hive and non-Hive Spark deployments.

BaseSessionStateBuilder holds properties that (together with newBuilder) are used to create a SessionState.

Name Description

analyzer

Logical analyzer

catalog

SessionCatalog

Used to create Analyzer, Optimizer and a SessionState itself

Note	HiveSessionStateBuilder manages its own Hive-aware HiveSessionCatalog.

conf

SQLConf

customOperatorOptimizationRules

Custom operator optimization rules to add to the base Operator Optimization batch.

When requested for the custom rules, customOperatorOptimizationRules simply requests the SparkSessionExtensions to buildOptimizerRules.

experimentalMethods

ExperimentalMethods

extensions

SparkSessionExtensions

functionRegistry

FunctionRegistry

listenerManager

ExecutionListenerManager

optimizer

SparkOptimizer (that is downcast to the base Optimizer) that is created with the SessionCatalog and the ExperimentalMethods.

Note that the SparkOptimizer adds the customOperatorOptimizationRules to the operator optimization rules.

optimizer is used when BaseSessionStateBuilder is requested to create a SessionState (for the optimizerBuilder function to create an Optimizer when requested for the Optimizer).

planner

SparkPlanner

resourceLoader

SessionResourceLoader

sqlParser

ParserInterface

streamingQueryManager

Spark Structured Streaming’s StreamingQueryManager

udfRegistration

UDFRegistration

Note

BaseSessionStateBuilder defines a type alias NewBuilder for a function to create a BaseSessionStateBuilder.



type NewBuilder = (SparkSession, Option[SessionState]) => BaseSessionStateBuilder

type NewBuilder = (SparkSession, Option[SessionState]) => BaseSessionStateBuilder

Note	`BaseSessionStateBuilder` is an experimental and unstable API.

Creating Function to Build SessionState — `createClone` Method



createClone: (SparkSession, SessionState) => SessionState

createClone: (SparkSession, SessionState) => SessionState

createClone gives a function of SparkSession and SessionState that executes newBuilder followed by build.

Note	`createClone` is used exclusively when `BaseSessionStateBuilder` is requested for a SessionState

Creating SessionState Instance — `build` Method



build(): SessionState

build(): SessionState

build creates a SessionState with the following:

SharedState of SparkSession
SQLConf
ExperimentalMethods
FunctionRegistry
UDFRegistration
SessionCatalog
ParserInterface
Analyzer
Optimizer
SparkPlanner
StreamingQueryManager
ExecutionListenerManager
SessionResourceLoader
createQueryExecution
createClone

Note	`build` is used when: `SparkSession` is requested for the SessionState (and builds it using a class name per spark.sql.catalogImplementation configuration property `BaseSessionStateBuilder` is requested to create a clone of a `SessionState`

Creating BaseSessionStateBuilder Instance

BaseSessionStateBuilder takes the following when created:

SparkSession
Optional SessionState

Getting Function to Create QueryExecution For LogicalPlan — `createQueryExecution` Method



createQueryExecution: LogicalPlan => QueryExecution

createQueryExecution: LogicalPlan => QueryExecution

createQueryExecution simply returns a function that takes a LogicalPlan and creates a QueryExecution with the SparkSession and the logical plan.

Note	`createQueryExecution` is used exclusively when `BaseSessionStateBuilder` is requested to create a SessionState instance.

BaseSessionStateBuilder — Generic Builder of SessionState