spark技术分享MetadataLog — Contract for Metadata Storage
MetadataLog is the contract to store metadata.
MetadataLog Contract
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
package org.apache.spark.sql.execution.streaming trait MetadataLog[T] { def add(batchId: Long, metadata: T): Boolean def get(batchId: Long): Option[T] def get(startId: Option[Long], endId: Option[Long]): Array[(Long, T)] def getLatest(): Option[(Long, T)] def purge(thresholdBatchId: Long): Unit } |
| Method | Description | ||
|---|---|---|---|
|
Retrieves the latest-committed batch with the metadata if available from the metadata storage.
|
|||
