spark-sql-spark技术分享-第9页

Logging

2013-05-12admin阅读(4099)

Logging

Spark uses log4j for logging.

Logging Levels

The valid logging levels are log4j’s Levels (from most specific to least):

OFF (most specific, no logging)
FATAL (most specific, little data)
ERROR
WARN
INFO
DEBUG
TRACE (least specific, a lot of data)
ALL (least specific, all data)

conf/log4j.properties

You can set up the default logging for Spark shell in conf/log4j.properties. Use conf/log4j.properties.template as a starting point.

Setting Default Log Level Programatically

Refer to Setting Default Log Level Programatically in SparkContext — Entry Point to Spark Core.

Setting Log Levels in Spark Applications

In standalone Spark applications or while in Spark Shell session, use the following:



import org.apache.log4j.{Level, Logger}

Logger.getLogger(classOf[RackResolver]).getLevel
Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)

1

2

3

4

5

6

7

8

9

import org.apache.log4j.{Level, Logger}

Logger.getLogger(classOf[RackResolver]).getLevel

Logger.getLogger("org").setLevel(Level.OFF)

Logger.getLogger("akka").setLevel(Level.OFF)

sbt

When running a Spark application from within sbt using run task, you can use the following build.sbt to configure logging levels:



fork in run := true
javaOptions in run ++= Seq(
  "-Dlog4j.debug=true",
  "-Dlog4j.configuration=log4j.properties")
outputStrategy := Some(StdoutOutput)

1

2

3

4

5

6

7

8

9

fork in run := true

javaOptions in run ++= Seq(

"-Dlog4j.debug=true",

"-Dlog4j.configuration=log4j.properties")

outputStrategy := Some(StdoutOutput)

With the above configuration log4j.properties file should be on CLASSPATH which can be in src/main/resources directory (that is included in CLASSPATH by default).

When run starts, you should see the following output in sbt:



[spark-activator]> run
[info] Running StreamingApp
log4j: Trying to find [log4j.properties] using context classloader sun.misc.Launcher$AppClassLoader@1b6d3586.
log4j: Using URL [file:/Users/jacek/dev/oss/spark-activator/target/scala-2.11/classes/log4j.properties] for automatic log4j configuration.
log4j: Reading configuration from URL file:/Users/jacek/dev/oss/spark-activator/target/scala-2.11/classes/log4j.properties

1

2

3

4

5

6

7

8

9

[spark-activator]> run

[info] Running StreamingApp

log4j: Trying to find [log4j.properties] using context classloader sun.misc.Launcher$AppClassLoader@1b6d3586.

log4j: Using URL [file:/Users/jacek/dev/oss/spark-activator/target/scala-2.11/classes/log4j.properties] for automatic log4j configuration.

log4j: Reading configuration from URL file:/Users/jacek/dev/oss/spark-activator/target/scala-2.11/classes/log4j.properties

Disabling Logging

Use the following conf/log4j.properties to disable logging completely:



log4j.logger.org=OFF

1

2

3

4

5

log4j.logger.org=OFF

BasicWriteJobStatsTracker

2013-05-11admin阅读(3469)

BasicWriteJobStatsTracker

BasicWriteJobStatsTracker is a concrete WriteJobStatsTracker.

BasicWriteJobStatsTracker is created when DataWritingCommand and Spark Structured Streaming’s FileStreamSink are requested for one.

When requested for a new WriteTaskStatsTracker, BasicWriteJobStatsTracker creates a new BasicWriteTaskStatsTracker.

Creating BasicWriteJobStatsTracker Instance

BasicWriteJobStatsTracker takes the following when created:

Serializable Hadoop Configuration
Metrics (Map[String, SQLMetric])

WriteJobStatsTracker

2013-05-10admin阅读(1396)

WriteJobStatsTracker

WriteJobStatsTracker is the abstraction of WriteJobStatsTrackers that can create a new WriteTaskStatsTracker and processStats.

Method Description

newTaskInstance



newTaskInstance(): WriteTaskStatsTracker

1

2

3

4

5

newTaskInstance(): WriteTaskStatsTracker

Creates a new WriteTaskStatsTracker

Used when EmptyDirectoryWriteTask, SingleDirectoryWriteTask and DynamicPartitionWriteTask are requested for the statsTrackers

processStats



processStats(stats: Seq[WriteTaskStats]): Unit

1

2

3

4

5

processStats(stats: Seq[WriteTaskStats]): Unit

Used when…FIXME

Note	BasicWriteJobStatsTracker is the one and only known implementation of the WriteJobStatsTracker Contract in Apache Spark.

BasicWriteTaskStatsTracker

2013-05-09admin阅读(3363)

BasicWriteTaskStatsTracker

BasicWriteTaskStatsTracker is a concrete WriteTaskStatsTracker.

BasicWriteTaskStatsTracker is created exclusively when BasicWriteJobStatsTracker is requested for one.

BasicWriteTaskStatsTracker takes a Hadoop Configuration when created.

Getting Final WriteTaskStats — `getFinalStats` Method



getFinalStats(): WriteTaskStats

1

2

3

4

5

getFinalStats(): WriteTaskStats

Note	`getFinalStats` is part of the WriteTaskStatsTracker Contract to get the final WriteTaskStats statistics computed so far.

getFinalStats…FIXME

WriteTaskStatsTracker

2013-05-08admin阅读(3096)

WriteTaskStatsTracker

WriteTaskStatsTracker is the abstraction of WriteTaskStatsTrackers that collect the statistics of the number of buckets, files, partitions and rows processed.

Method Description

getFinalStats



getFinalStats(): WriteTaskStats

1

2

3

4

5

getFinalStats(): WriteTaskStats

The final WriteTaskStats statistics computed so far

Used when EmptyDirectoryWriteTask, SingleDirectoryWriteTask and DynamicPartitionWriteTask are requested to execute

newBucket



newBucket(bucketId: Int): Unit

1

2

3

4

5

newBucket(bucketId: Int): Unit

Used when…FIXME

newFile



newFile(filePath: String): Unit

1

2

3

4

5

newFile(filePath: String): Unit

Used when…FIXME

newPartition



newPartition(partitionValues: InternalRow): Unit

1

2

3

4

5

newPartition(partitionValues: InternalRow): Unit

Used when…FIXME

newRow



newRow(row: InternalRow): Unit

1

2

3

4

5

newRow(row: InternalRow): Unit

Used when…FIXME

Note	BasicWriteTaskStatsTracker is the one and only known implementation of the WriteTaskStatsTracker Contract in Apache Spark.

BasicWriteTaskStats

2013-05-07admin阅读(1240)

BasicWriteTaskStats

BasicWriteTaskStats is a basic WriteTaskStats that carries the following statistics:

numPartitions
numFiles
numBytes
numRows

BasicWriteTaskStats is created exclusively when BasicWriteTaskStatsTracker is requested for getFinalStats.

WriteTaskStats

2013-05-06admin阅读(1193)

WriteTaskStats

WriteTaskStats is the no-method contract of data statistics collected during a Write Task.

Note	BasicWriteTaskStats is the one and only known implementation of the WriteTaskStats Contract in Apache Spark.

SQLAppStatusStore

2013-05-05admin阅读(1568)

SQLAppStatusStore

SQLAppStatusStore is…FIXME

SQLAppStatusStore is created when SQLAppStatusListener or SQLAppStatusPlugin create a SQLTab.

Creating SQLAppStatusStore Instance

SQLAppStatusStore takes the following when created:

KVStore
Optional SQLAppStatusListener

SQLAppStatusPlugin

2013-05-04admin阅读(1482)

SQLAppStatusPlugin

SQLAppStatusPlugin is a AppStatusPlugin…FIXME

`setupUI` Method



setupUI(ui: SparkUI): Unit

1

2

3

4

5

setupUI(ui: SparkUI): Unit

Note	`setupUI` is part of AppStatusPlugin Contract to…FIXME.

setupUI…FIXME

SQLAppStatusListener Spark Listener

2013-05-03admin阅读(2056)

SQLAppStatusListener Spark Listener

SQLAppStatusListener is a SparkListener that…FIXME

Table 1. SQLAppStatusListener’s Internal Properties (e.g. Registries, Counters and Flags)
Name	Description
`liveUpdatePeriodNs`
`liveExecutions`
`stageMetrics`
`uiInitialized`

`onExecutionStart` Internal Method



onExecutionStart(event: SparkListenerSQLExecutionStart): Unit

1

2

3

4

5

onExecutionStart(event: SparkListenerSQLExecutionStart): Unit

onExecutionStart…FIXME

Note	`onExecutionStart` is used exclusively when `SQLAppStatusListener` handles a SparkListenerSQLExecutionStart event.

`onJobStart` Callback



onJobStart(event: SparkListenerJobStart): Unit

1

2

3

4

5

onJobStart(event: SparkListenerJobStart): Unit

Note	`onJobStart` is part of SparkListener Contract to…FIXME

onJobStart…FIXME

`onStageSubmitted` Callback



onStageSubmitted(event: SparkListenerStageSubmitted): Unit

1

2

3

4

5

onStageSubmitted(event: SparkListenerStageSubmitted): Unit

Note	`onStageSubmitted` is part of SparkListener Contract to…FIXME

onStageSubmitted…FIXME

`onJobEnd` Callback



onJobEnd(event: SparkListenerJobEnd): Unit

1

2

3

4

5

onJobEnd(event: SparkListenerJobEnd): Unit

Note	`onJobEnd` is part of SparkListener Contract to…FIXME

onJobEnd…FIXME

`onExecutorMetricsUpdate` Callback



onExecutorMetricsUpdate(event: SparkListenerExecutorMetricsUpdate): Unit

1

2

3

4

5

onExecutorMetricsUpdate(event: SparkListenerExecutorMetricsUpdate): Unit

Note	`onExecutorMetricsUpdate` is part of SparkListener Contract to…FIXME

onExecutorMetricsUpdate…FIXME

`onTaskEnd` Callback



onTaskEnd(event: SparkListenerTaskEnd): Unit

1

2

3

4

5

onTaskEnd(event: SparkListenerTaskEnd): Unit

Note	`onTaskEnd` is part of SparkListener Contract to…FIXME

onTaskEnd…FIXME

Handling SparkListenerEvent — `onOtherEvent` Callback



onOtherEvent(event: SparkListenerEvent): Unit

1

2

3

4

5

onOtherEvent(event: SparkListenerEvent): Unit

Note	`onOtherEvent` is part of SparkListener Contract to…FIXME

onOtherEvent…FIXME

spark-sql 第9页

Logging

Logging Levels

conf/log4j.properties

Setting Default Log Level Programatically

Setting Log Levels in Spark Applications

sbt

Disabling Logging

BasicWriteJobStatsTracker

Creating BasicWriteJobStatsTracker Instance

WriteJobStatsTracker

BasicWriteTaskStatsTracker

Getting Final WriteTaskStats — getFinalStats Method

WriteTaskStatsTracker

BasicWriteTaskStats

WriteTaskStats

SQLAppStatusStore

Creating SQLAppStatusStore Instance

SQLAppStatusPlugin

setupUI Method

SQLAppStatusListener Spark Listener

onExecutionStart Internal Method

onJobStart Callback

onStageSubmitted Callback

onJobEnd Callback

onExecutorMetricsUpdate Callback

onTaskEnd Callback

Handling SparkListenerEvent — onOtherEvent Callback

欢迎关注：spark技术分享

关注公众号：spark技术分享

QQ咨询

回顶部

Getting Final WriteTaskStats — `getFinalStats` Method

`setupUI` Method

`onExecutionStart` Internal Method

`onJobStart` Callback

`onStageSubmitted` Callback

`onJobEnd` Callback

`onExecutorMetricsUpdate` Callback

`onTaskEnd` Callback

Handling SparkListenerEvent — `onOtherEvent` Callback