MicroBatchReadSupport Contract — Data Sources with MicroBatchReaders

2015-07-16 分类：structured-streaming 阅读(2649) 评论(0)

MicroBatchReadSupport is the abstraction of data sources with a MicroBatchReader.

ContinuousReadSupport defines a single createMicroBatchReader method that creates a MicroBatchReader.



MicroBatchReader createMicroBatchReader(
  Optional<StructType> schema,
  String checkpointLocation,
  DataSourceOptions options)

MicroBatchReader createMicroBatchReader(

Optional<StructType> schema,

String checkpointLocation,

DataSourceOptions options)

createMicroBatchReader is used when:

MicroBatchExecution is requested for the analyzed logical plan (and creates a StreamingExecutionRelation for a StreamingRelationV2 with a MicroBatchReadSupport data source)
DataStreamReader is requested to create a streaming query for a MicroBatchReadSupport data source

赞(0) 打赏

关注公众号：spark技术分享