spark技术分享SchemaRelationProvider Contract — Relation Providers With Mandatory User-Defined Schema
SchemaRelationProvider is the contract of BaseRelation providers that require a user-defined schema while creating a relation.
The requirement of specifying a user-defined schema is enforced when DataSource is requested for a BaseRelation for a given data source format. If not specified, DataSource throws a AnalysisException:
|
1 2 3 4 5 |
A schema needs to be specified when using [className]. |
|
1 2 3 4 5 6 7 8 9 10 11 12 |
package org.apache.spark.sql.sources trait SchemaRelationProvider { def createRelation( sqlContext: SQLContext, parameters: Map[String, String], schema: StructType): BaseRelation } |
| Method | Description |
|---|---|
|
|
Creates a BaseRelation for the user-defined schema Used exclusively when |
|
Note
|
There are no known direct implementation of PrunedFilteredScan Contract in Spark SQL. |
|
Tip
|
Use RelationProvider for data source providers with schema inference. |
|
Tip
|
Use both SchemaRelationProvider and RelationProvider if a data source should support both schema inference and user-defined schemas.
|
