关注 spark技术分享,
撸spark源码 玩spark最佳实践

JDBCOptions — JDBC Data Source Options

JDBCOptions — JDBC Data Source Options

JDBCOptions represents the options of the JDBC data source.

Table 1. Options for JDBC Data Source
Option / Key Default Value Description

batchsize

1000

The minimum value is 1

Used exclusively when JdbcRelationProvider is requested to write the rows of a structured query (a DataFrame) to a table through JdbcUtils helper object and its saveTable.

createTableColumnTypes

Used exclusively when JdbcRelationProvider is requested to write the rows of a structured query (a DataFrame) to a table through JdbcUtils helper object and its createTable.

createTableOptions

Empty string

Used exclusively when JdbcRelationProvider is requested to write the rows of a structured query (a DataFrame) to a table through JdbcUtils helper object and its createTable.

customSchema

(undefined)

Specifies the custom data types of the read schema (that is used at load time)

customSchema is a comma-separated list of field definitions with column names and their data types in a canonical SQL representation, e.g. id DECIMAL(38, 0), name STRING.

customSchema defines the data types of the columns that will override the data types inferred from the table schema and follows the following pattern:

Used exclusively when JDBCRelation is requested for the schema.

dbtable

(required)

Used when:

driver

(recommended) Class name of the JDBC driver to use

Used exclusively when JDBCOptions is created. When the driver option is defined, the JDBC driver class will get registered with Java’s java.sql.DriverManager.

Note
driver takes precedence over the class name of the driver for the url option.

After the JDBC driver class was registered, the driver class is used exclusively when JdbcUtils helper object is requested to createConnectionFactory.

fetchsize

0

Hint to the JDBC driver as to the number of rows that should be fetched from the database when more rows are needed for ResultSet objects generated by a Statement

The minimum value is 0 (which tells the JDBC driver to do the estimates)

Used exclusively when JDBCRDD is requested to compute a partition.

isolationLevel

READ_UNCOMMITTED

One of the following:

  • NONE

  • READ_UNCOMMITTED

  • READ_COMMITTED

  • REPEATABLE_READ

  • SERIALIZABLE

Used exclusively when JdbcUtils is requested to saveTable.

lowerBound

Lower bound of partition column

Used exclusively when JdbcRelationProvider is requested to create a BaseRelation for reading

numPartitions

Number of partitions to use for loading or saving data

Used when:

partitionColumn

Name of the column used to partition dataset (using a JDBCPartitioningInfo).

Used exclusively when JdbcRelationProvider is requested to create a BaseRelation for reading (with proper JDBCPartitions with WHERE clause)

When defined, the lowerBound, upperBound and numPartitions options are also required.

When undefined, lowerBound and upperBound have to be undefined.

truncate

false

(used only for writing) Enables table truncation

Used exclusively when JdbcRelationProvider is requested to write the rows of a structured query (a DataFrame) to a table

sessionInitStatement

A generic SQL statement (or PL/SQL block) executed before reading a table/query

Used exclusively when JDBCRDD is requested to compute a partition.

upperBound

Upper bound of the partition column

Used exclusively when JdbcRelationProvider is requested to create a BaseRelation for reading

url

(required) A JDBC URL to use to connect to a database

Note
The options are case-insensitive.

JDBCOptions is created when:

Creating JDBCOptions Instance

JDBCOptions takes the following when created:

  • JDBC URL

  • Name of the table

  • Case-insensitive configuration parameters (i.e. Map[String, String])

The input URL and table are set as the current url and dbtable options (overriding the values in the input parameters if defined).

Converting Parameters (Options) to Java Properties — asProperties Property

asProperties…​FIXME

Note

asProperties is used when:

asConnectionProperties Property

asConnectionProperties…​FIXME

Note
asConnectionProperties is used exclusively when JdbcUtils is requested to createConnectionFactory
赞(0) 打赏
未经允许不得转载:spark技术分享 » JDBCOptions — JDBC Data Source Options
分享到: 更多 (0)

关注公众号:spark技术分享

联系我们联系我们

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏