StreamingExecutionRelation Leaf Logical Operator for Streaming Source At Execution-spark技术分享

StreamingExecutionRelation Leaf Logical Operator for Streaming Source At Execution

StreamingExecutionRelation is a leaf logical operator (i.e. LogicalPlan) that represents a streaming source in the logical query plan of a streaming Dataset.

The main use of StreamingExecutionRelation logical operator is to be a “placeholder” in a logical query plan that will be replaced with the real relation (with new data that has arrived since the last batch) or an empty LocalRelation when StreamExecution runs a single streaming batch.

StreamingExecutionRelation is created for a StreamingRelation in analyzed logical query plan (that is the execution representation of a streaming Dataset).

Note	Right after `StreamExecution` has started running streaming batches it initializes the streaming sources by transforming the analyzed logical plan of the streaming Dataset so that every StreamingRelation logical operator is replaced by the corresponding `StreamingExecutionRelation`.

Figure 1. StreamingExecutionRelation Represents Streaming Source At Execution

Note	`StreamingExecutionRelation` is also resolved (aka planned) to a StreamingRelationExec physical operator in StreamingRelationStrategy execution planning strategy only when explaining a streaming `Dataset`.

Creating StreamingExecutionRelation Instance

StreamingExecutionRelation takes the following when created:

Streaming source
Output attributes

Creating StreamingExecutionRelation (for MemoryStream Source) — `apply` Factory Method



apply(source: Source): StreamingExecutionRelation

apply(source: Source): StreamingExecutionRelation

apply creates a StreamingExecutionRelation for the input source and with the attributes of the schema of the source.

Note	`apply` is used exclusively when `MemoryStream` is created (and the logical plan initialized).

StreamingExecutionRelation Leaf Logical Operator for Streaming Source At Execution