Utils Helper Object-spark技术分享

`getLocalDir` Method



getLocalDir(conf: SparkConf): String

1

2

3

4

5

getLocalDir(conf: SparkConf): String

getLocalDir…FIXME

Note

getLocalDir is used when:

Utils is requested to fetchFile
SparkEnv is created (on the driver)
spark-shell is launched
Spark on YARN’s Client is requested to prepareLocalResources and create __spark_conf__.zip archive with configuration files and Spark configuration
PySpark’s PythonBroadcast is requested to readObject
PySpark’s EvalPythonExec is requested to doExecute

`fetchFile` Method



fetchFile(
  url: String,
  targetDir: File,
  conf: SparkConf,
  securityMgr: SecurityManager,
  hadoopConf: Configuration,
  timestamp: Long,
  useCache: Boolean): File

1

2

3

4

5

6

7

8

9

10

11

12

fetchFile(

url: String,

targetDir: File,

conf: SparkConf,

securityMgr: SecurityManager,

hadoopConf: Configuration,

timestamp: Long,

useCache: Boolean): File

fetchFile…FIXME

Note	`fetchFile` is used when: `SparkContext` is requested to addFile `Executor` is requested to updateDependencies Spark Standalone’s `DriverRunner` is requested to `downloadUserJar`

`getOrCreateLocalRootDirsImpl` Internal Method



getOrCreateLocalRootDirsImpl(conf: SparkConf): Array[String]

1

2

3

4

5

getOrCreateLocalRootDirsImpl(conf: SparkConf): Array[String]

getOrCreateLocalRootDirsImpl…FIXME

Note	`getOrCreateLocalRootDirsImpl` is used exclusively when `Utils` is requested to getOrCreateLocalRootDirs

`getOrCreateLocalRootDirs` Internal Method



getOrCreateLocalRootDirs(conf: SparkConf): Array[String]

1

2

3

4

5

getOrCreateLocalRootDirs(conf: SparkConf): Array[String]

getOrCreateLocalRootDirs…FIXME

Note	`getOrCreateLocalRootDirs` is used when: `Utils` is requested to getLocalDir `Worker` is requested to handle a LaunchExecutor message

Utils Helper Object

Utils Helper Object

`getLocalDir` Method

`fetchFile` Method

`getOrCreateLocalRootDirsImpl` Internal Method

`getOrCreateLocalRootDirs` Internal Method

相关推荐

欢迎关注：spark技术分享

热门标签

近期文章

分类目录

关注公众号：spark技术分享

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏

QQ咨询

回顶部

Utils Helper Object

getLocalDir Method

fetchFile Method

getOrCreateLocalRootDirsImpl Internal Method

getOrCreateLocalRootDirs Internal Method

相关推荐

欢迎关注：spark技术分享

热门标签

近期文章

分类目录

关注公众号：spark技术分享

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏

QQ咨询

回顶部

`getLocalDir` Method

`fetchFile` Method

`getOrCreateLocalRootDirsImpl` Internal Method

`getOrCreateLocalRootDirs` Internal Method