spark技术分享Continuous Stream Processing (Structured Streaming V2)
Continuous Stream Processing is a new stream processing model in Spark Structured Streaming (often referred as Structured Streaming V2) that is used for Trigger.Continuous trigger.
Continuous stream processing uses ContinuousExecution as the stream execution engine.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
import org.apache.spark.sql.streaming.Trigger import scala.concurrent.duration._ val sq = spark .readStream .format("rate") .load .writeStream .format("console") .option("truncate", false) .trigger(Trigger.Continuous(10.seconds)) // <-- Uses ContinuousExecution for execution .queryName("rate2console") .start assert(sq.isActive) scala> sq.explain == Physical Plan == WriteToContinuousDataSource ConsoleWriter[numRows=20, truncate=false] +- *(1) Project [timestamp#758, value#759L] +- *(1) ScanV2 rate[timestamp#758, value#759L] // sq.stop |
When DataStreamReader is requested to create a streaming query for a ContinuousReadSupport data source, it creates…FIXME
