spark技术分享StreamSourceProvider — Streaming Data Source Provider
StreamSourceProvider is the contract for objects that can create a streaming data source for a format (e.g. text file) or system (e.g. Apache Kafka) by their short names.
StreamSourceProvider is used when DataSource is requested for the name and schema of a streaming source or just creates one.
| Name | Description |
|---|---|
|
Creates KafkaSourceProvider for |
|
|
Creates TextSocketSources for |
StreamSourceProvider Contract
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
trait StreamSourceProvider { def sourceSchema( sqlContext: SQLContext, schema: Option[StructType], providerName: String, parameters: Map[String, String]): (String, StructType) def createSource( sqlContext: SQLContext, metadataPath: String, schema: Option[StructType], providerName: String, parameters: Map[String, String]): Source } |
|
Note
|
StreamSourceProvider is an experimental contract.
|
| Method | Description | ||||
|---|---|---|---|---|---|
|
Creates a streaming source for a format or system (to continually read data).
Used exclusively when Spark SQL’s
|
|||||
|
Defines the name and the schema of a streaming source |
