spark技术分享Micro-Batch Stream Processing (Structured Streaming V1)
Micro-Batch Stream Processing is a stream processing model in Spark Structured Streaming (often referred as Structured Streaming V1) that is used for Trigger.Once and Trigger.ProcessingTime triggers.
Micro-batch stream processing uses MicroBatchExecution as the stream execution engine.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
import org.apache.spark.sql.streaming.Trigger import scala.concurrent.duration._ val sq = spark .readStream .format("rate") .load .writeStream .format("console") .option("truncate", false) .trigger(Trigger.ProcessingTime(1.minute)) // <-- Uses MicroBatchExecution for execution .queryName("rate2console") .start assert(sq.isActive) scala> sq.explain == Physical Plan == WriteToDataSourceV2 org.apache.spark.sql.execution.streaming.sources.MicroBatchWriter@678e6267 +- *(1) Project [timestamp#54, value#55L] +- *(1) ScanV2 rate[timestamp#54, value#55L] // sq.stop |
