spark技术分享RelationProvider Contract — Relation Providers With Schema Inference
RelationProvider is the contract of BaseRelation providers that create a relation with schema inference.
|
Note
|
Schema inference is also called schema discovery. |
The requirement of not specifying a user-defined schema or having one that does not match the relation is enforced when DataSource is requested for a BaseRelation for a given data source format. If specified and does not match, DataSource throws a AnalysisException:
|
1 2 3 4 5 |
[className] does not allow user-specified schemas. |
|
1 2 3 4 5 6 7 8 9 10 11 |
package org.apache.spark.sql.sources trait RelationProvider { def createRelation( sqlContext: SQLContext, parameters: Map[String, String]): BaseRelation } |
| Method | Description |
|---|---|
|
|
Creates a BaseRelation for loading data from an external data source Used exclusively when |
| RelationProvider | Description |
|---|---|
|
Tip
|
Use SchemaRelationProvider for relation providers that require a user-defined schema. |
