KafkaSourceRDDOffsetRange

2012-05-08 分类：spark-sql 阅读(1359) 评论(0)

KafkaSourceRDDOffsetRange is an offset range that one KafkaSourceRDDPartition partition of a KafkaSourceRDD has to read.

KafkaSourceRDDOffsetRange is created when:

KafkaRelation is requested to build a distributed data scan with column pruning (as a TableScan) (and creates a KafkaSourceRDD)
KafkaSourceRDD is requested to resolveRange
(Spark Structured Streaming) KafkaSource is requested to getBatch

KafkaSourceRDDOffsetRange takes the following when created:

Note	TopicPartition is a topic name and partition number.

赞(0) 打赏

关注公众号：spark技术分享