spark技术分享LogicalRDD — Logical Scan Over RDD
LogicalRDD is a leaf logical operator with MultiInstanceRelation support for a logical representation of a scan over RDD of internal binary rows.
LogicalRDD is created when:
-
Datasetis requested to checkpoint -
SparkSessionis requested to create a DataFrame from an RDD of internal binary rows
|
Note
|
LogicalRDD is resolved to RDDScanExec when BasicOperators execution planning strategy is executed.
|
newInstance Method
|
1 2 3 4 5 |
newInstance(): LogicalRDD.this.type |
|
Note
|
newInstance is part of MultiInstanceRelation Contract to…FIXME.
|
newInstance…FIXME
Computing Statistics — computeStats Method
|
1 2 3 4 5 |
computeStats(): Statistics |
|
Note
|
computeStats is part of LeafNode Contract to compute statistics for cost-based optimizer.
|
computeStats…FIXME
Creating LogicalRDD Instance
LogicalRDD takes the following when created:
-
Output schema attributes
-
RDDof internal binary rows
