DataSourceReader-spark技术分享

DataSourceReader

DataSourceReader is the contract for data source readers with a custom DataReaderFactory.



package org.apache.spark.sql.sources.v2.reader;

interface DataSourceReader {
  StructType readSchema();
  List<DataReaderFactory<Row>> createDataReaderFactories();
}

package org.apache.spark.sql.sources.v2.reader;

interface DataSourceReader {

StructType readSchema();

List<DataReaderFactory<Row>> createDataReaderFactories();

}

Note	`DataSourceReader` is an `Evolving` contract that is evolving towards becoming a stable API, but is not a stable API yet and can change from one feature release to another release. In other words, using the contract is as treading on thin ice.

Table 1. DataSourceReader Contract
Method	Description
`readSchema`	Used when…FIXME
`createDataReaderFactories`	Used when…FIXME

Table 2. DataSourceReaders
DataSourceReader	Description
`ContinuousReader`	Used in Spark Structured Streaming
`MicroBatchReader`	Used in Spark Structured Streaming
`SupportsPushDownCatalystFilters`
`SupportsPushDownFilters`
`SupportsPushDownRequiredColumns`
`SupportsReportPartitioning`
`SupportsReportStatistics`
`SupportsScanColumnarBatch`
`SupportsScanUnsafeRow`

DataSourceReader

DataSourceReader

相关推荐

欢迎关注：spark技术分享

热门标签

近期文章

分类目录

关注公众号：spark技术分享

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏

QQ咨询

回顶部