spark技术分享JDBCRDD
JDBCRDD is a RDD of internal binary rows that represents a structured query over a table in a database accessed via JDBC.
|
Note
|
JDBCRDD represents a “SELECT requiredColumns FROM table” query.
|
JDBCRDD is created exclusively when JDBCRDD is requested to scanTable (when JDBCRelation is requested to build a scan).
| Name | Description |
|---|---|
|
|
Used when…FIXME |
|
|
Filters as a SQL Used when…FIXME |
Computing Partition (in TaskContext) — compute Method
|
1 2 3 4 5 |
compute(thePart: Partition, context: TaskContext): Iterator[InternalRow] |
|
Note
|
compute is part of Spark Core’s RDD Contract to compute a partition (in a TaskContext).
|
compute…FIXME
resolveTable Method
|
1 2 3 4 5 |
resolveTable(options: JDBCOptions): StructType |
resolveTable…FIXME
|
Note
|
resolveTable is used exclusively when JDBCRelation is requested for the schema.
|
Creating RDD for Distributed Data Scan — scanTable Object Method
|
1 2 3 4 5 6 7 8 9 10 11 |
scanTable( sc: SparkContext, schema: StructType, requiredColumns: Array[String], filters: Array[Filter], parts: Array[Partition], options: JDBCOptions): RDD[InternalRow] |
scanTable takes the url option.
scanTable finds the corresponding JDBC dialect (per the url option) and requests it to quote the column identifiers in the input requiredColumns.
scanTable uses the JdbcUtils object to createConnectionFactory and prune columns from the input schema to include the input requiredColumns only.
In the end, scanTable creates a new JDBCRDD.
|
Note
|
scanTable is used exclusively when JDBCRelation is requested to build a distributed data scan with column pruning and filter pushdown.
|
Creating JDBCRDD Instance
JDBCRDD takes the following when created:
-
Schema (StructType)
-
Array of Filter predicates
JDBCRDD initializes the internal registries and counters.
getPartitions Method
|
1 2 3 4 5 |
getPartitions: Array[Partition] |
|
Note
|
getPartitions is part of Spark Core’s RDD Contract to…FIXME
|
getPartitions simply returns the partitions (this JDBCRDD was created with).
pruneSchema Internal Method
|
1 2 3 4 5 |
pruneSchema(schema: StructType, columns: Array[String]): StructType |
pruneSchema…FIXME
|
Note
|
pruneSchema is used when…FIXME
|
Converting Filter Predicate to SQL Expression — compileFilter Object Method
|
1 2 3 4 5 |
compileFilter(f: Filter, dialect: JdbcDialect): Option[String] |
compileFilter…FIXME
|
Note
|
|
