TextSocketSourceProvider-spark技术分享

TextSocketSourceProvider

TextSocketSourceProvider is a StreamSourceProvider for TextSocketSource that read records from host and port.

TextSocketSourceProvider is a DataSourceRegister, too.

The short name of the data source is socket.

It requires two mandatory options (that you can set using option method):

host which is the host name.
port which is the port number. It must be an integer.

TextSocketSourceProvider also supports includeTimestamp option that is a boolean flag that you can use to include timestamps in the schema.

includeTimestamp Option

Caution

FIXME

createSource

createSource grabs the two mandatory options — host and port — and returns an TextSocketSource.

sourceSchema

sourceSchema returns textSocket as the name of the source and the schema that can be one of the two available schemas:

SCHEMA_REGULAR (default) which is a schema with a single value field of String type.
SCHEMA_TIMESTAMP when includeTimestamp flag option is set. It is not, i.e. false, by default. The schema are value field of StringType type and timestamp field of TimestampType type of format yyyy-MM-dd HH:mm:ss.

Tip	Read about schema.

Internally, it starts by printing out the following WARN message to the logs:



WARN TextSocketSourceProvider: The socket source should not be used for production applications! It does not support recovery and stores state indefinitely.

WARN TextSocketSourceProvider: The socket source should not be used for production applications! It does not support recovery and stores state indefinitely.

It then checks whether host and port parameters are defined and if not it throws a AnalysisException:



Set a host to read from with option("host", ...).

Set a host to read from with option("host", ...).

TextSocketSourceProvider