Generate-spark技术分享

Generate Unary Logical Operator for Lateral Views

Generate is a unary logical operator that is created to represent the following (after a logical plan is analyzed):

Generator or GeneratorOuter expressions (by ExtractGenerator logical evaluation rule)
SQL’s LATERAL VIEW clause (in SELECT or FROM clauses)

resolved flag is…FIXME

Note	`resolved` is part of LogicalPlan Contract to…FIXME.

producedAttributes…FIXME

The output schema of a Generate is…FIXME

Note	`Generate` logical operator is resolved to GenerateExec unary physical operator in BasicOperators execution planning strategy.

Tip

Use generate operator from Catalyst DSL to create a Generate logical operator, e.g. for testing or Spark SQL internals exploration.



import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.types._
val lr = LocalRelation('key.int, 'values.array(StringType))

// JsonTuple generator
import org.apache.spark.sql.catalyst.expressions.JsonTuple
import org.apache.spark.sql.catalyst.dsl.expressions._
import org.apache.spark.sql.catalyst.expressions.Expression
val children: Seq[Expression] = Seq("e")
val json_tuple = JsonTuple(children)

import org.apache.spark.sql.catalyst.dsl.plans._  // <-- gives generate
val plan = lr.generate(
  generator = json_tuple,
  join = true,
  outer = true,
  alias = Some("alias"),
  outputNames = Seq.empty)
scala> println(plan.numberedTreeString)
00 'Generate json_tuple(e), true, true, alias
01 +- LocalRelation <empty>, [key#0, values#1]

import org.apache.spark.sql.catalyst.plans.logical._

import org.apache.spark.sql.types._

val lr = LocalRelation('key.int, 'values.array(StringType))

// JsonTuple generator

import org.apache.spark.sql.catalyst.expressions.JsonTuple

import org.apache.spark.sql.catalyst.dsl.expressions._

import org.apache.spark.sql.catalyst.expressions.Expression

val children: Seq[Expression] = Seq("e")

val json_tuple = JsonTuple(children)

import org.apache.spark.sql.catalyst.dsl.plans._ // <-- gives generate

val plan = lr.generate(

generator = json_tuple,

join = true,

outer = true,

alias = Some("alias"),

outputNames = Seq.empty)

scala> println(plan.numberedTreeString)

00 'Generate json_tuple(e), true, true, alias

01 +- LocalRelation <empty>, [key#0, values#1]

Creating Generate Instance

Generate takes the following when created:

Generator expression
join flag…FIXME
outer flag…FIXME
Optional qualifier
Output attributes
Child logical plan

Generate initializes the internal registries and counters.

Generate

Generate Unary Logical Operator for Lateral Views

Creating Generate Instance

相关推荐

欢迎关注：spark技术分享

热门标签

近期文章

分类目录

关注公众号：spark技术分享

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏

QQ咨询

回顶部