spark技术分享StreamingRelation Leaf Logical Operator for Streaming Source
StreamingRelation is a leaf logical operator (i.e. LogicalPlan) that represents a streaming source in a logical plan.
StreamingRelation is created when DataStreamReader is requested to load data from a streaming source and creates a streaming Dataset.
Figure 1. StreamingRelation Represents Streaming Source
|
1 2 3 4 5 6 7 8 9 10 |
val rate = spark. readStream. // <-- creates a DataStreamReader format("rate"). load("hello") // <-- creates a StreamingRelation scala> println(rate.queryExecution.logical.numberedTreeString) 00 StreamingRelation DataSource(org.apache.spark.sql.SparkSession@4e5dcc50,rate,List(),None,List(),None,Map(path -> hello),None), rate, [timestamp#0, value#1L] |
isStreaming is always enabled (i.e. true).
|
1 2 3 4 5 6 7 8 |
import org.apache.spark.sql.execution.streaming.StreamingRelation val relation = rate.queryExecution.logical.asInstanceOf[StreamingRelation] scala> relation.isStreaming res1: Boolean = true |
toString gives the source name.
|
1 2 3 4 5 6 |
scala> println(relation) rate |
|
Note
|
StreamingRelation is resolved (aka planned) to StreamingExecutionRelation (right after StreamExecution starts running batches).
|
Creating StreamingRelation for DataSource — apply Factory Method
|
1 2 3 4 5 |
apply(dataSource: DataSource): StreamingRelation |
apply creates a StreamingRelation for the input streaming DataSource and the short name and the schema of the streaming source (behind the DataSource).
|
Note
|
apply creates a StreamingRelation logical operator (for the input DataSource) that represents a streaming source.
|
|
Note
|
apply is used exclusively when DataStreamReader is requested to load data from a streaming source to a streaming Dataset.
|
