ContinuousReader Contract — DataSourceReaders For Continuous Stream Processing-spark技术分享

ContinuousReader Contract — DataSourceReaders For Continuous Stream Processing

ContinuousReader is the extension of the DataSourceReader (and BaseStreamingSource) contracts for data source readers that can read in a continuous processing mode.

Method Description

commit



void commit(Offset end)

void commit(Offset end)

Used exclusively when ContinuousExecution is requested to commit an epoch

deserializeOffset



Offset deserializeOffset(String json)

Offset deserializeOffset(String json)

Used when ContinuousExecution is requested to run a streaming query and commit an epoch

getStartOffset



Offset getStartOffset()

Offset getStartOffset()

Used when…FIXME

mergeOffsets



Offset mergeOffsets(PartitionOffset[] offsets)

Offset mergeOffsets(PartitionOffset[] offsets)

Used when…FIXME

needsReconfiguration



boolean needsReconfiguration()

boolean needsReconfiguration()

Used when…FIXME

setStartOffset



void setStartOffset(Optional<Offset> start)

void setStartOffset(Optional<Offset> start)

Used when…FIXME

Table 2. ContinuousReaders
ContinuousReader	Description
ContinuousMemoryStream
KafkaContinuousReader
RateStreamContinuousReader
TextSocketContinuousReader