spark技术分享KafkaSourceOffset
KafkaSourceOffset is a OffsetV2 that…FIXME
KafkaSourceOffset takes a collection of Kafka TopicPartitions and their offsets when created.
Getting Partition Offsets — getPartitionOffsets Method
|
1 2 3 4 5 |
getPartitionOffsets(offset: Offset): Map[TopicPartition, Long] |
getPartitionOffsets takes KafkaSourceOffset.partitionToOffsets from offset.
If offset is KafkaSourceOffset, getPartitionOffsets takes the partitions and offsets straight from it.
If however offset is SerializedOffset, getPartitionOffsets deserializes the offsets from JSON.
getPartitionOffsets reports an IllegalArgumentException when offset is neither KafkaSourceOffset or SerializedOffset.
|
1 2 3 4 5 |
Invalid conversion from offset of [class] to KafkaSourceOffset |
|
Note
|
getPartitionOffsets is used exclusively when KafkaSource generates a DataFrame with records from Kafka for a batch.
|
