spark技术分享InsertIntoHadoopFsRelationCommand Logical Command
InsertIntoHadoopFsRelationCommand is a DataWritingCommand that writes the data of the query out using the FileFormat.
InsertIntoHadoopFsRelationCommand is created when:
-
DataSourceis requested to planForWritingFileFormat -
DataSourceAnalysis post-hoc logical resolution rule is executed (and resolves a InsertIntoTable logical operator with a HadoopFsRelation)
Executing Logical Command — run Method
|
1 2 3 4 5 |
run(session: SparkSession): Seq[Row] |
|
Note
|
run is part of RunnableCommand Contract to execute (run) a logical command.
|
run uses the spark.sql.hive.manageFilesourcePartitions configuration property to…FIXME
|
Caution
|
FIXME When is the catalogTable defined? |
|
Caution
|
FIXME When is tracksPartitionsInCatalog of CatalogTable enabled?
|
run…FIXME
run uses SchemaUtils to make sure that there are no duplicates in the outputColumnNames.
Creating InsertIntoHadoopFsRelationCommand Instance
InsertIntoHadoopFsRelationCommand takes the following when created:
-
Output Hadoop’s Path
|
Note
|
staticPartitions may hold zero or more partitions as follows:
With that, staticPartitions are simply the partitions of an InsertIntoTable logical operator. |
