spark技术分享StreamingRelationStrategy Execution Planning Strategy for StreamingRelation and StreamingExecutionRelation Logical Operators
StreamingRelationStrategy is an streaming execution planning strategy (i.e. Strategy) that converts StreamingRelation and StreamingExecutionRelation logical operators (in the logical query plan of a streaming Dataset) to StreamingRelationExec physical operator.
|
Note
|
StreamingRelation logical operator represents a streaming source in a logical plan and is created when StreamingExecutionRelation logical operator also represents a streaming source in a logical plan, but is used internally when |
StreamingRelationStrategy is used exclusively when IncrementalExecution plans the logical plan of a streaming Dataset for explain operator.
StreamingRelationStrategy converts StreamingRelation and StreamingExecutionRelation logical operators in a logical query plan to a StreamingRelationExec physical operator (with their sourceName and output schema) to give a corresponding physical query plan.
StreamingRelationStrategy is available using SessionState (of a SparkSession).
|
1 2 3 4 5 |
spark.sessionState.planner.StreamingRelationStrategy |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
val rates = spark. readStream. format("rate"). load // <-- gives a streaming Dataset with a logical plan with StreamingRelation logical operator // StreamingRelation logical operator for the rate streaming source scala> println(rates.queryExecution.logical.numberedTreeString) 00 StreamingRelation DataSource(org.apache.spark.sql.SparkSession@31ba0af0,rate,List(),None,List(),None,Map(),None), rate, [timestamp#0, value#1L] // StreamingRelationExec physical operator (shown without "Exec" suffix) scala> rates.explain == Physical Plan == StreamingRelation rate, [timestamp#0, value#1L] // Let's do the planning manually import spark.sessionState.planner.StreamingRelationStrategy val physicalPlan = StreamingRelationStrategy.apply(rates.queryExecution.logical).head scala> println(physicalPlan.numberedTreeString) 00 StreamingRelation rate, [timestamp#0, value#1L] |
