RDD Dependencies-spark技术分享

RDD Dependencies

Dependency class is the base (abstract) class to model a dependency relationship between two or more RDDs.

Dependency has a single method rdd to access the RDD that is behind a dependency.



def rdd: RDD[T]

def rdd: RDD[T]

Whenever you apply a transformation (e.g. map, flatMap) to a RDD you build the so-called RDD lineage graph. Dependency-ies represent the edges in a lineage graph.

Note	NarrowDependency and ShuffleDependency are the two top-level subclasses of `Dependency` abstract class.

Table 1. Kinds of Dependencies
Name	Description
NarrowDependency
ShuffleDependency
OneToOneDependency
PruneDependency
RangeDependency

Note

The dependencies of a RDD are available using dependencies method.



// A demo RDD
scala> val myRdd = sc.parallelize(0 to 9).groupBy(_ % 2)
myRdd: org.apache.spark.rdd.RDD[(Int, Iterable[Int])] = ShuffledRDD[8] at groupBy at <console>:24

scala> myRdd.foreach(println)
(0,CompactBuffer(0, 2, 4, 6, 8))
(1,CompactBuffer(1, 3, 5, 7, 9))

scala> myRdd.dependencies
res5: Seq[org.apache.spark.Dependency[_]] = List(org.apache.spark.ShuffleDependency@27ace619)

// Access all RDDs in the demo RDD lineage
scala> myRdd.dependencies.map(_.rdd).foreach(println)
MapPartitionsRDD[7] at groupBy at <console>:24

// A demo RDD

scala> val myRdd = sc.parallelize(0 to 9).groupBy(_ % 2)

myRdd: org.apache.spark.rdd.RDD[(Int, Iterable[Int])] = ShuffledRDD[8] at groupBy at <console>:24

scala> myRdd.foreach(println)

(0,CompactBuffer(0, 2, 4, 6, 8))

(1,CompactBuffer(1, 3, 5, 7, 9))

scala> myRdd.dependencies

res5: Seq[org.apache.spark.Dependency[_]] = List(org.apache.spark.ShuffleDependency@27ace619)

// Access all RDDs in the demo RDD lineage

scala> myRdd.dependencies.map(_.rdd).foreach(println)

MapPartitionsRDD[7] at groupBy at <console>:24

You use toDebugString method to print out the RDD lineage in a user-friendly way.



scala> myRdd.toDebugString
res6: String =
(8) ShuffledRDD[8] at groupBy at <console>:24 []
 +-(8) MapPartitionsRDD[7] at groupBy at <console>:24 []
    |  ParallelCollectionRDD[6] at parallelize at <console>:24 []

scala> myRdd.toDebugString

res6: String =

(8) ShuffledRDD[8] at groupBy at <console>:24 []

+-(8) MapPartitionsRDD[7] at groupBy at <console>:24 []

| ParallelCollectionRDD[6] at parallelize at <console>:24 []

RDD Dependencies

RDD Dependencies

相关推荐

欢迎关注：spark技术分享

热门标签

近期文章

分类目录

关注公众号：spark技术分享

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏

QQ咨询

回顶部