DataWritingCommand Contract — Logical Commands That Write Data-spark技术分享

DataWritingCommand Contract — Logical Commands That Write Data

DataWritingCommand is an extension of the Command contract for logical commands that write query data to a relation when executed.

DataWritingCommand is resolved to a DataWritingCommandExec physical operator when BasicOperators execution planning strategy is executed (i.e. plan a logical plan to a physical plan).

Property Description

outputColumnNames



outputColumnNames: Seq[String]

outputColumnNames: Seq[String]

The output column names of the analyzed input query plan

Used when DataWritingCommand is requested for the outputColumns

query



query: LogicalPlan

query: LogicalPlan

The logical query plan representing the data to write (i.e. whose result will be inserted into a relation)

Used when DataWritingCommand is requested for the child nodes and outputColumns.

run



run(sparkSession: SparkSession, child: SparkPlan): Seq[Row]

run(sparkSession: SparkSession, child: SparkPlan): Seq[Row]

Executes the command

Used when:

DataWritingCommandExec physical operator is requested for the sideEffectResult
DataSource is requested to write data to a data source per save mode followed by reading rows back (when CreateDataSourceTableAsSelectCommand logical command is executed)

When requested for the child nodes, DataWritingCommand simply returns the logical query plan.

DataWritingCommand defines custom performance metrics.

Table 2. DataWritingCommand’s Performance Metrics
Key	Name (in web UI)	Description
`numFiles`	number of written files
`numOutputBytes`	bytes of written output
`numOutputRows`	number of output rows
`numParts`	number of dynamic part

The performance metrics are used when:

DataWritingCommand is requested for the BasicWriteJobStatsTracker
DataWritingCommandExec physical operator is requested for the metrics

Table 3. DataWritingCommands (Direct Implementations)
DataWritingCommand	Description
CreateDataSourceTableAsSelectCommand
CreateHiveTableAsSelectCommand
InsertIntoHadoopFsRelationCommand
SaveAsHiveFile	Contract for commands that write query result as Hive files

`basicWriteJobStatsTracker` Method



basicWriteJobStatsTracker(hadoopConf: Configuration): BasicWriteJobStatsTracker

basicWriteJobStatsTracker(hadoopConf: Configuration): BasicWriteJobStatsTracker

basicWriteJobStatsTracker simply creates and returns a new BasicWriteJobStatsTracker (with the given Hadoop Configuration and the metrics).

Note	`basicWriteJobStatsTracker` is used when: `SaveAsHiveFile` is requested to saveAsHiveFile (when InsertIntoHiveDirCommand and InsertIntoHiveTable logical commands are executed) InsertIntoHadoopFsRelationCommand logical command is executed

Output Columns — `outputColumns` Method



outputColumns: Seq[Attribute]

outputColumns: Seq[Attribute]

outputColumns…FIXME

Note	`outputColumns` is used when: CreateHiveTableAsSelectCommand, InsertIntoHiveDirCommand and InsertIntoHadoopFsRelationCommand logical commands are executed `SaveAsHiveFile` is requested to saveAsHiveFile

DataWritingCommand Contract — Logical Commands That Write Data