ClientDistributedCacheManager-spark技术分享

ClientDistributedCacheManager

ClientDistributedCacheManager is a mere wrapper to hold the collection of cache-related resource entries CacheEntry (as distCacheEntries) to add resources to and later update Spark configuration with files to distribute.

Caution

FIXME What is a resource? Is this a file only?

Adding Cache-Related Resource (addResource method)



addResource(
  fs: FileSystem,
  conf: Configuration,
  destPath: Path,
  localResources: HashMap[String, LocalResource],
  resourceType: LocalResourceType,
  link: String,
  statCache: Map[URI, FileStatus],
  appMasterOnly: Boolean = false): Unit

addResource(

fs: FileSystem,

conf: Configuration,

destPath: Path,

localResources: HashMap[String, LocalResource],

resourceType: LocalResourceType,

link: String,

statCache: Map[URI, FileStatus],

appMasterOnly: Boolean = false): Unit

Updating Spark Configuration with Resources to Distribute (updateConfiguration method)



updateConfiguration(conf: SparkConf): Unit

updateConfiguration(conf: SparkConf): Unit

updateConfiguration sets the following internal Spark configuration settings in the input conf Spark configuration:

It uses the internal distCacheEntries with resources to distribute.

Note	It is later used in `ApplicationMaster` when it prepares local resources.

ClientDistributedCacheManager