spark技术分享Debugging Spark
Using spark-shell and IntelliJ IDEA
Start spark-shell with SPARK_SUBMIT_OPTS environment variable that configures the JVM’s JDWP.
|
1 2 3 4 5 |
SPARK_SUBMIT_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005" ./bin/spark-shell |
Attach IntelliJ IDEA to the JVM process using Run > Attach to Local Process menu.
Using sbt
Use sbt -jvm-debug 5005, connect to the remote JVM at the port 5005 using IntelliJ IDEA, place breakpoints on the desired lines of the source code of Spark.
|
1 2 3 4 5 6 7 |
➜ sparkme-app sbt -jvm-debug 5005 Listening for transport dt_socket at address: 5005 ... |
Run Spark context and the breakpoints get triggered.
|
1 2 3 4 5 6 |
scala> val sc = new SparkContext(conf) 15/11/14 22:58:46 INFO SparkContext: Running Spark version 1.6.0-SNAPSHOT |
|
Tip
|
Read Debugging chapter in IntelliJ IDEA 15.0 Help. |
