FileFormatWriter Helper Object-spark技术分享

FileFormatWriter Helper Object

FileFormatWriter is a Scala object that allows for writing the result of a structured query.

Tip

Enable ERROR, INFO, DEBUG logging level for org.apache.spark.sql.execution.datasources.FileFormatWriter logger to see what happens inside.

Add the following line to conf/log4j.properties:



log4j.logger.org.apache.spark.sql.execution.datasources.FileFormatWriter=INFO

log4j.logger.org.apache.spark.sql.execution.datasources.FileFormatWriter=INFO

Refer to Logging.

Writing Query Result — `write` Method



write(
  sparkSession: SparkSession,
  plan: SparkPlan,
  fileFormat: FileFormat,
  committer: FileCommitProtocol,
  outputSpec: OutputSpec,
  hadoopConf: Configuration,
  partitionColumns: Seq[Attribute],
  bucketSpec: Option[BucketSpec],
  statsTrackers: Seq[WriteJobStatsTracker],
  options: Map[String, String]): Set[String]

write(

sparkSession: SparkSession,

plan: SparkPlan,

fileFormat: FileFormat,

committer: FileCommitProtocol,

outputSpec: OutputSpec,

hadoopConf: Configuration,

partitionColumns: Seq[Attribute],

bucketSpec: Option[BucketSpec],

statsTrackers: Seq[WriteJobStatsTracker],

options: Map[String, String]): Set[String]

write…FIXME

Note

write is used when:

SaveAsHiveFile is requested to saveAsHiveFile (when InsertIntoHiveDirCommand and InsertIntoHiveTable logical commands are executed)
InsertIntoHadoopFsRelationCommand logical command is executed
Structured Streaming’s FileStreamSink is requested to add a streaming batch (addBatch)

`executeTask` Internal Method



executeTask(
  description: WriteJobDescription,
  sparkStageId: Int,
  sparkPartitionId: Int,
  sparkAttemptNumber: Int,
  committer: FileCommitProtocol,
  iterator: Iterator[InternalRow]): WriteTaskResult

executeTask(

description: WriteJobDescription,

sparkStageId: Int,

sparkPartitionId: Int,

sparkAttemptNumber: Int,

committer: FileCommitProtocol,

iterator: Iterator[InternalRow]): WriteTaskResult

executeTask…FIXME

Note	`executeTask` is used exclusively when `FileFormatWriter` is requested to write the result of a structured query.

`processStats` Internal Method



processStats(
  statsTrackers: Seq[WriteJobStatsTracker],
  statsPerTask: Seq[Seq[WriteTaskStats]]): Unit

processStats(

statsTrackers: Seq[WriteJobStatsTracker],

statsPerTask: Seq[Seq[WriteTaskStats]]): Unit

processStats…FIXME

Note	`processStats` is used exclusively when `FileFormatWriter` is requested to write the result of a structured query.

FileFormatWriter Helper Object