DataSource — Pluggable Data Source-spark技术分享

DataSource — Pluggable Data Source

DataSource is…FIXME

DataSource is created when…FIXME

Tip	Read DataSource — Pluggable Data Sources (for Spark SQL’s batch structured queries).

Table 1. DataSource’s Internal Properties (e.g. Registries, Counters and Flags)
Name	Description
`providingClass`	java.lang.Class that corresponds to the className (that can be a fully-qualified class name or an alias of the data source)
`sourceInfo`	`SourceInfo` with the name, the schema, and optional partitioning columns of a source. Used when: `DataSource` creates a FileStreamSource (that requires the schema and the optional partitioning columns) `StreamingRelation` is created (for a `DataSource`)

Describing Name and Schema of Streaming Source — `sourceSchema` Internal Method



sourceSchema(): SourceInfo

sourceSchema(): SourceInfo

sourceSchema…FIXME

Note	`sourceSchema` is used exclusively when `DataSource` is requested SourceInfo.

Creating DataSource Instance

DataSource takes the following when created:

SparkSession
className, i.e. the fully-qualified class name or an alias of the data source
Paths (default: Nil, i.e. an empty collection)
Optional user-defined schema (default: None)
Names of the partition columns (default: empty)
Optional BucketSpec (default: None)
Configuration options (default: empty)
Optional CatalogTable (default: None)

DataSource initializes the internal registries and counters.

`createSource` Method



createSource(metadataPath: String): Source

createSource(metadataPath: String): Source

createSource…FIXME

Note	`createSource` is used when…FIXME

Creating Streaming Sink — `createSink` Method



createSink(outputMode: OutputMode): Sink

createSink(outputMode: OutputMode): Sink

createSink creates a streaming sink for StreamSinkProvider or FileFormat data sources.

Tip	Find out more on `FileFormat` data sources in FileFormat — Data Sources to Read and Write Data In Files section in The Internals of Spark SQL book.

Internally, createSink creates a new instance of the providingClass and branches off per type:

For a StreamSinkProvider, createSink simply delegates the call and requests it to create a streaming sink
For a FileFormat, createSink creates a FileStreamSink when path option is specified and the output mode is Append

createSink throws a IllegalArgumentException when path option is not specified for a FileFormat data source:



'path' is not specified

'path' is not specified

createSink throws an AnalysisException when the given OutputMode is different from Append for a FileFormat data source:



Data source [className] does not support [outputMode] output mode

Data source [className] does not support [outputMode] output mode

createSink throws an UnsupportedOperationException for unsupported data source formats:



Data source [className] does not support streamed writing

Data source [className] does not support streamed writing

Note	`createSink` is used exclusively when `DataStreamWriter` is requested to create and start a streaming query.

DataSource — Pluggable Data Source