AbstractApplicationResource-spark技术分享

AbstractApplicationResource

AbstractApplicationResource is a BaseAppResource with a set of URI paths that are common across implementations.



// start spark-shell
$ http http://localhost:4040/api/v1/applications
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 257
Content-Type: application/json
Date: Tue, 05 Jun 2018 18:46:32 GMT
Server: Jetty(9.3.z-SNAPSHOT)
Vary: Accept-Encoding, User-Agent

[
    {
        "attempts": [
            {
                "appSparkVersion": "2.3.1-SNAPSHOT",
                "completed": false,
                "duration": 0,
                "endTime": "1969-12-31T23:59:59.999GMT",
                "endTimeEpoch": -1,
                "lastUpdated": "2018-06-05T15:04:48.328GMT",
                "lastUpdatedEpoch": 1528211088328,
                "sparkUser": "jacek",
                "startTime": "2018-06-05T15:04:48.328GMT",
                "startTimeEpoch": 1528211088328
            }
        ],
        "id": "local-1528211089216",
        "name": "Spark shell"
    }
]

$ http http://localhost:4040/api/v1/applications/local-1528211089216/storage/rdd
HTTP/1.1 200 OK
Content-Length: 3
Content-Type: application/json
Date: Tue, 05 Jun 2018 18:48:00 GMT
Server: Jetty(9.3.z-SNAPSHOT)
Vary: Accept-Encoding, User-Agent

[]

// Execute the following query in spark-shell
spark.range(5).cache.count

$ http http://localhost:4040/api/v1/applications/local-1528211089216/storage/rdd
// output omitted for brevity

// start spark-shell

$ http http://localhost:4040/api/v1/applications

HTTP/1.1 200 OK

Content-Encoding: gzip

Content-Length: 257

Content-Type: application/json

Date: Tue, 05 Jun 2018 18:46:32 GMT

Server: Jetty(9.3.z-SNAPSHOT)

Vary: Accept-Encoding, User-Agent

[

{

"attempts": [

{

"appSparkVersion": "2.3.1-SNAPSHOT",

"completed": false,

"duration": 0,

"endTime": "1969-12-31T23:59:59.999GMT",

"endTimeEpoch": -1,

"lastUpdated": "2018-06-05T15:04:48.328GMT",

"lastUpdatedEpoch": 1528211088328,

"sparkUser": "jacek",

"startTime": "2018-06-05T15:04:48.328GMT",

"startTimeEpoch": 1528211088328

}

"id": "local-1528211089216",

"name": "Spark shell"

}

]

$ http http://localhost:4040/api/v1/applications/local-1528211089216/storage/rdd

HTTP/1.1 200 OK

Content-Length: 3

Content-Type: application/json

Date: Tue, 05 Jun 2018 18:48:00 GMT

Server: Jetty(9.3.z-SNAPSHOT)

Vary: Accept-Encoding, User-Agent

[]

// Execute the following query in spark-shell

spark.range(5).cache.count

$ http http://localhost:4040/api/v1/applications/local-1528211089216/storage/rdd

// output omitted for brevity

Table 1. AbstractApplicationResources
AbstractApplicationResource	Description
OneApplicationResource	Handles `applications/appId` requests
OneApplicationAttemptResource

Table 2. AbstractApplicationResource’s Paths
Path	HTTP Method	Description
`allexecutors`	GET	allExecutorList
`environment`	GET	environmentInfo
`executors`	GET	executorList
`jobs`	GET	jobsList
`jobs/{jobId: \\d+}`	GET	oneJob
`logs`	GET	getEventLogs
`stages`		stages
`storage/rdd/{rddId: \\d+}`	GET	rddData
`storage/rdd`	GET	rddList

`rddList` Method



rddList(): Seq[RDDStorageInfo]

rddList(): Seq[RDDStorageInfo]

rddList…FIXME

Note	`rddList` is used when…FIXME

`environmentInfo` Method



environmentInfo(): ApplicationEnvironmentInfo

environmentInfo(): ApplicationEnvironmentInfo

environmentInfo…FIXME

Note	`environmentInfo` is used when…FIXME

`rddData` Method



rddData(@PathParam("rddId") rddId: Int): RDDStorageInfo

rddData(@PathParam("rddId") rddId: Int): RDDStorageInfo

rddData…FIXME

Note	`rddData` is used when…FIXME

`allExecutorList` Method



allExecutorList(): Seq[ExecutorSummary]

allExecutorList(): Seq[ExecutorSummary]

allExecutorList…FIXME

Note	`allExecutorList` is used when…FIXME

`executorList` Method



executorList(): Seq[ExecutorSummary]

executorList(): Seq[ExecutorSummary]

executorList…FIXME

Note	`executorList` is used when…FIXME

`oneJob` Method



oneJob(@PathParam("jobId") jobId: Int): JobData

oneJob(@PathParam("jobId") jobId: Int): JobData

oneJob…FIXME

Note	`oneJob` is used when…FIXME

`jobsList` Method



jobsList(@QueryParam("status") statuses: JList[JobExecutionStatus]): Seq[JobData]

jobsList(@QueryParam("status") statuses: JList[JobExecutionStatus]): Seq[JobData]

jobsList…FIXME

Note	`jobsList` is used when…FIXME

AbstractApplicationResource