WriteTaskStatsTracker-spark技术分享

WriteTaskStatsTracker

WriteTaskStatsTracker is the abstraction of WriteTaskStatsTrackers that collect the statistics of the number of buckets, files, partitions and rows processed.

Method Description

getFinalStats



getFinalStats(): WriteTaskStats

getFinalStats(): WriteTaskStats

The final WriteTaskStats statistics computed so far

Used when EmptyDirectoryWriteTask, SingleDirectoryWriteTask and DynamicPartitionWriteTask are requested to execute

newBucket



newBucket(bucketId: Int): Unit

newBucket(bucketId: Int): Unit

Used when…FIXME

newFile



newFile(filePath: String): Unit

newFile(filePath: String): Unit

Used when…FIXME

newPartition



newPartition(partitionValues: InternalRow): Unit

newPartition(partitionValues: InternalRow): Unit

Used when…FIXME

newRow



newRow(row: InternalRow): Unit

newRow(row: InternalRow): Unit

Used when…FIXME

Note	BasicWriteTaskStatsTracker is the one and only known implementation of the WriteTaskStatsTracker Contract in Apache Spark.