CodegenContext-spark技术分享

CodegenContext

CodegenContext is…FIXME

CodegenContext takes no input parameters.



import org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext
val ctx = new CodegenContext

import org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext

val ctx = new CodegenContext

CodegenContext is created when:

WholeStageCodegenExec physical operator is requested to generate a Java source code for the child operator (when WholeStageCodegenExec is executed)
CodeGenerator is requested for a new CodegenContext
GenerateUnsafeRowJoiner is requested for a UnsafeRowJoiner

CodegenContext stores expressions that don’t support codegen.


Example of CodegenContext.subexpressionElimination (through CodegenContext.generateExpressions)

import org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext
val ctx = new CodegenContext

// Use Catalyst DSL
import org.apache.spark.sql.catalyst.dsl.expressions._
val expressions = "hello".expr.as("world") :: "hello".expr.as("world") :: Nil

// FIXME Use a real-life query to extract the expressions

// CodegenContext.subexpressionElimination (where the elimination all happens) is a private method
// It is used exclusively in CodegenContext.generateExpressions which is public
// and does the elimination when it is enabled

// Note the doSubexpressionElimination flag is on
// Triggers the subexpressionElimination private method
ctx.generateExpressions(expressions, doSubexpressionElimination = true)

// subexpressionElimination private method uses ctx.equivalentExpressions
val commonExprs = ctx.equivalentExpressions.getAllEquivalentExprs

assert(commonExprs.length > 0, "No common expressions found")

Example of CodegenContext.subexpressionElimination (through CodegenContext.generateExpressions)

import org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext

val ctx = new CodegenContext

// Use Catalyst DSL

import org.apache.spark.sql.catalyst.dsl.expressions._

val expressions = "hello".expr.as("world") :: "hello".expr.as("world") :: Nil

// FIXME Use a real-life query to extract the expressions

// CodegenContext.subexpressionElimination (where the elimination all happens) is a private method

// It is used exclusively in CodegenContext.generateExpressions which is public

// and does the elimination when it is enabled

// Note the doSubexpressionElimination flag is on

// Triggers the subexpressionElimination private method

ctx.generateExpressions(expressions, doSubexpressionElimination = true)

// subexpressionElimination private method uses ctx.equivalentExpressions

val commonExprs = ctx.equivalentExpressions.getAllEquivalentExprs

assert(commonExprs.length > 0, "No common expressions found")

Table 1. CodegenContext’s Internal Properties (e.g. Registries, Counters and Flags)
Name	Description
`classFunctions`	Mutable Scala `Map` with function names, their Java source code and a class name New entries are added when `CodegenContext` is requested to addClass and addNewFunctionToClass Used when `CodegenContext` is requested to declareAddedFunctions
`equivalentExpressions`	EquivalentExpressions Expressions are added and then fetched as equivalent sets when `CodegenContext` is requested to subexpressionElimination (for generateExpressions with subexpression elimination enabled)
`currentVars`	The list of generated columns as input of current operator
`INPUT_ROW`	The variable name of the input row of the current operator
`placeHolderToComments`	Placeholders and their comments Used when…FIXME
`references`	References that are used to generate classes in the following code generators: GenerateMutableProjection GenerateOrdering GeneratePredicate GenerateSafeProjection GenerateUnsafeProjection WholeStageCodegenExec Elements are added when: `CodegenContext` is requested to addReferenceObj `CodegenFallback` is requested to doGenCode
`subExprEliminationExprs`	`SubExprEliminationStates` by Expression Used when…FIXME
`subexprFunctions`	Names of the functions that…FIXME

Generating Java Source Code For Code-Generated Evaluation of Multiple Expressions (With Optional Subexpression Elimination) — `generateExpressions` Method



generateExpressions(
  expressions: Seq[Expression],
  doSubexpressionElimination: Boolean = false): Seq[ExprCode]

generateExpressions(

expressions: Seq[Expression],

doSubexpressionElimination: Boolean = false): Seq[ExprCode]

(only with subexpression elimination enabled) generateExpressions does subexpressionElimination of the input expressions.

In the end, generateExpressions requests every expressions to generate the Java source code for code-generated (non-interpreted) expression evaluation.

Note

generateExpressions is used when:

GenerateMutableProjection is requested to create a MutableProjection
GenerateUnsafeProjection is requested to create an ExprCode for Catalyst expressions
HashAggregateExec is requested to generate the Java source code for whole-stage consume path with grouping keys

`addReferenceObj` Method



addReferenceObj(objName: String, obj: Any, className: String = null): String

addReferenceObj(objName: String, obj: Any, className: String = null): String

addReferenceObj…FIXME

Note	`addReferenceObj` is used when…FIXME

`subexpressionEliminationForWholeStageCodegen` Method



subexpressionEliminationForWholeStageCodegen(expressions: Seq[Expression]): SubExprCodes

subexpressionEliminationForWholeStageCodegen(expressions: Seq[Expression]): SubExprCodes

subexpressionEliminationForWholeStageCodegen…FIXME

Note	`subexpressionEliminationForWholeStageCodegen` is used exclusively when `HashAggregateExec` is requested to generate a Java source code for whole-stage consume path (with grouping keys or not).

Adding Function to Generated Class — `addNewFunction` Method



addNewFunction(
  funcName: String,
  funcCode: String,
  inlineToOuterClass: Boolean = false): String

addNewFunction(

funcName: String,

funcCode: String,

inlineToOuterClass: Boolean = false): String

addNewFunction…FIXME

Note	`addNewFunction` is used when…FIXME

`subexpressionElimination` Internal Method



subexpressionElimination(expressions: Seq[Expression]): Unit

subexpressionElimination(expressions: Seq[Expression]): Unit

subexpressionElimination requests EquivalentExpressions to addExprTree for every expression (in the input expressions).

subexpressionElimination requests EquivalentExpressions for the equivalent sets of expressions with at least two equivalent expressions (aka common expressions).

For every equivalent expression set, subexpressionElimination does the following:

Takes the first expression and requests it to generate a Java source code for the expression tree
addNewFunction and adds it to subexprFunctions
Creates a SubExprEliminationState and adds it with every common expression in the equivalent expression set to subExprEliminationExprs

Note	`subexpressionElimination` is used exclusively when `CodegenContext` is requested to generateExpressions (with subexpression elimination enabled).

Adding Mutable State — `addMutableState` Method



addMutableState(
  javaType: String,
  variableName: String,
  initFunc: String => String = _ => "",
  forceInline: Boolean = false,
  useFreshName: Boolean = true): String

addMutableState(

javaType: String,

variableName: String,

initFunc: String => String = _ => "",

forceInline: Boolean = false,

useFreshName: Boolean = true): String

addMutableState…FIXME



val input = ctx.addMutableState("scala.collection.Iterator", "input", v => s"$v = inputs[0];")

val input = ctx.addMutableState("scala.collection.Iterator", "input", v => s"$v = inputs[0];")

Note	`addMutableState` is used when…FIXME

Adding Immutable State (Unless Exists Already) — `addImmutableStateIfNotExists` Method



addImmutableStateIfNotExists(
  javaType: String,
  variableName: String,
  initFunc: String => String = _ => ""): Unit

addImmutableStateIfNotExists(

javaType: String,

variableName: String,

initFunc: String => String = _ => ""): Unit

addImmutableStateIfNotExists…FIXME



val ctx: CodegenContext = ???
val partitionMaskTerm = "partitionMask"
ctx.addImmutableStateIfNotExists(ctx.JAVA_LONG, partitionMaskTerm)

val ctx: CodegenContext = ???

val partitionMaskTerm = "partitionMask"

ctx.addImmutableStateIfNotExists(ctx.JAVA_LONG, partitionMaskTerm)

Note	`addImmutableStateIfNotExists` is used when…FIXME

`freshName` Method



freshName(name: String): String

freshName(name: String): String

freshName…FIXME

Note	`freshName` is used when…FIXME

`addNewFunctionToClass` Internal Method



addNewFunctionToClass(
  funcName: String,
  funcCode: String,
  className: String): mutable.Map[String, mutable.Map[String, String]]

addNewFunctionToClass(

funcName: String,

funcCode: String,

className: String): mutable.Map[String, mutable.Map[String, String]]

addNewFunctionToClass…FIXME

Note	`addNewFunctionToClass` is used when…FIXME

`addClass` Internal Method



addClass(className: String, classInstance: String): Unit

addClass(className: String, classInstance: String): Unit

addClass…FIXME

Note	`addClass` is used when…FIXME

`declareAddedFunctions` Method



declareAddedFunctions(): String

declareAddedFunctions(): String

declareAddedFunctions…FIXME

Note	`declareAddedFunctions` is used when…FIXME

`declareMutableStates` Method



declareMutableStates(): String

declareMutableStates(): String

declareMutableStates…FIXME

Note	`declareMutableStates` is used when…FIXME

`initMutableStates` Method



initMutableStates(): String

initMutableStates(): String

initMutableStates…FIXME

Note	`initMutableStates` is used when…FIXME

`initPartition` Method



initPartition(): String

initPartition(): String

initPartition…FIXME

Note	`initPartition` is used when…FIXME

`emitExtraCode` Method



emitExtraCode(): String

emitExtraCode(): String

emitExtraCode…FIXME

Note	`emitExtraCode` is used when…FIXME

`addPartitionInitializationStatement` Method



addPartitionInitializationStatement(statement: String): Unit

addPartitionInitializationStatement(statement: String): Unit

addPartitionInitializationStatement…FIXME

Note	`addPartitionInitializationStatement` is used when…FIXME

CodegenContext

CodegenContext

Generating Java Source Code For Code-Generated Evaluation of Multiple Expressions (With Optional Subexpression Elimination) — `generateExpressions` Method

`addReferenceObj` Method

`subexpressionEliminationForWholeStageCodegen` Method

Adding Function to Generated Class — `addNewFunction` Method

`subexpressionElimination` Internal Method

Adding Mutable State — `addMutableState` Method

Adding Immutable State (Unless Exists Already) — `addImmutableStateIfNotExists` Method

`freshName` Method

`addNewFunctionToClass` Internal Method

`addClass` Internal Method

`declareAddedFunctions` Method

`declareMutableStates` Method

`initMutableStates` Method

`initPartition` Method

`emitExtraCode` Method

`addPartitionInitializationStatement` Method

相关推荐

欢迎关注：spark技术分享

热门标签

近期文章

分类目录

关注公众号：spark技术分享

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏

QQ咨询

回顶部

CodegenContext

Generating Java Source Code For Code-Generated Evaluation of Multiple Expressions (With Optional Subexpression Elimination) — generateExpressions Method

addReferenceObj Method

subexpressionEliminationForWholeStageCodegen Method

Adding Function to Generated Class — addNewFunction Method

subexpressionElimination Internal Method

Adding Mutable State — addMutableState Method

Adding Immutable State (Unless Exists Already) — addImmutableStateIfNotExists Method

freshName Method

addNewFunctionToClass Internal Method

addClass Internal Method

declareAddedFunctions Method

declareMutableStates Method

initMutableStates Method

initPartition Method

emitExtraCode Method

addPartitionInitializationStatement Method

相关推荐

欢迎关注：spark技术分享

热门标签

近期文章

分类目录

关注公众号：spark技术分享

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏

QQ咨询

回顶部

Generating Java Source Code For Code-Generated Evaluation of Multiple Expressions (With Optional Subexpression Elimination) — `generateExpressions` Method

`addReferenceObj` Method

`subexpressionEliminationForWholeStageCodegen` Method

Adding Function to Generated Class — `addNewFunction` Method

`subexpressionElimination` Internal Method

Adding Mutable State — `addMutableState` Method

Adding Immutable State (Unless Exists Already) — `addImmutableStateIfNotExists` Method

`freshName` Method

`addNewFunctionToClass` Internal Method

`addClass` Internal Method

`declareAddedFunctions` Method

`declareMutableStates` Method

`initMutableStates` Method

`initPartition` Method

`emitExtraCode` Method

`addPartitionInitializationStatement` Method