InterpretedProjection-spark技术分享

InterpretedProjection

InterpretedProjection is a Projection that…FIXME

InterpretedProjection takes expressions when created.



// HACK: Disable symbolToColumn implicit conversion
// It is imported automatically in spark-shell (and makes demos impossible)
// implicit def symbolToColumn(s: Symbol): org.apache.spark.sql.ColumnName
trait ThatWasABadIdea
implicit def symbolToColumn(ack: ThatWasABadIdea) = ack

import org.apache.spark.sql.catalyst.dsl.expressions._
val boundRef = 'hello.string.at(4)

import org.apache.spark.sql.catalyst.expressions.{Expression, Literal}
val expressions: Seq[Expression] = Seq(Literal(1), boundRef)

import org.apache.spark.sql.catalyst.expressions.InterpretedProjection
val ip = new InterpretedProjection(expressions)
scala> println(ip)
Row => [1,input[4, string, true]]

// HACK: Disable symbolToColumn implicit conversion

// It is imported automatically in spark-shell (and makes demos impossible)

// implicit def symbolToColumn(s: Symbol): org.apache.spark.sql.ColumnName

trait ThatWasABadIdea

implicit def symbolToColumn(ack: ThatWasABadIdea) = ack

import org.apache.spark.sql.catalyst.dsl.expressions._

val boundRef = 'hello.string.at(4)

import org.apache.spark.sql.catalyst.expressions.{Expression, Literal}

val expressions: Seq[Expression] = Seq(Literal(1), boundRef)

import org.apache.spark.sql.catalyst.expressions.InterpretedProjection

val ip = new InterpretedProjection(expressions)

scala> println(ip)

Row => [1,input[4, string, true]]

InterpretedProjection is created when:

UserDefinedGenerator is requested to initializeConverters
ConvertToLocalRelation logical optimization is executed (to transform Project logical operators)
HiveGenericUDTF is evaluated
ScriptTransformationExec physical operator is executed

Initializing Nondeterministic Expressions — `initialize` Method



initialize(partitionIndex: Int): Unit

initialize(partitionIndex: Int): Unit

Note	`initialize` is part of Projection Contract to…FIXME.

initialize requests Nondeterministic expressions (in expressions) to initialize with the partitionIndex.

InterpretedProjection