DataSourceV2-spark技术分享

DataSourceV2

DataSourceV2 is the contract of Data Sources V2 that FIXME.

DataSourceV2 defines no methods or values and acts as a marker interface.



package org.apache.spark.sql.sources.v2;

public interface DataSourceV2 {}

package org.apache.spark.sql.sources.v2;

public interface DataSourceV2 {}

Note	Implementations should mix-in at least one of the interfaces like `ReadSupport` and `WriteSupport`. Otherwise it’s simply a dummy data source which is un-readable and un-writable.

Note	`DataSourceV2` is an `Evolving` contract that is evolving towards becoming a stable API, but is not a stable API yet and can change from one feature release to another release. In other words, using the contract is as treading on thin ice.

Table 1. DataSourceV2s
DataSourceV2	Description
`ConsoleSinkProvider`	Used in Spark Structured Streaming
`ContinuousReadSupport`	Used in Spark Structured Streaming
`MemorySinkV2`	Used in Spark Structured Streaming
`MicroBatchReadSupport`	Used in Spark Structured Streaming
`RateSourceProvider`	Used in Spark Structured Streaming
`RateSourceProviderV2`	Used in Spark Structured Streaming
`ReadSupport`
`ReadSupportWithSchema`
`SessionConfigSupport`
`StreamWriteSupport`
`WriteSupport`