MicroBatchReader Contract — DataSourceReaders For Micro-Batch Stream Processing-spark技术分享

MicroBatchReader Contract — DataSourceReaders For Micro-Batch Stream Processing

MicroBatchReader is the extension of the DataSourceReader (and BaseStreamingSource) contracts for data source readers that can read in a micro-batch processing mode.

Method Description

commit



void commit(Offset end)

void commit(Offset end)

Used when…FIXME

deserializeOffset



Offset deserializeOffset(String json)

Offset deserializeOffset(String json)

Used when…FIXME

getEndOffset



Offset getEndOffset()

Offset getEndOffset()

Used when…FIXME

getStartOffset



Offset getStartOffset()

Offset getStartOffset()

Used when…FIXME

setOffsetRange



void setOffsetRange(
  Optional<Offset> start,
  Optional<Offset> end)

void setOffsetRange(

Optional<Offset> start,

Optional<Offset> end)

Used when…FIXME

Table 2. MicroBatchReaders
MicroBatchReader	Description
KafkaMicroBatchReader
MemoryStream
RateStreamMicroBatchReader
TextSocketMicroBatchReader