spark技术分享CodegenFallback Contract — Catalyst Expressions with Fallback Code Generation Mode
CodegenFallback is the contract of Catalyst expressions that do not support a Java code generation and want to fall back to interpreted mode (aka fallback mode).
CodegenFallback is used when CollapseCodegenStages physical optimization is requested to execute (and enforce whole-stage codegen requirements for Catalyst expressions).
|
1 2 3 4 5 6 7 8 9 |
package org.apache.spark.sql.catalyst.expressions.codegen trait CodegenFallback extends Expression { // No properties (vals and methods) that have no implementation } |
| CodegenFallback | Description |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
Example — CurrentTimestamp Expression with nullable flag disabled import org.apache.spark.sql.catalyst.expressions.CurrentTimestamp val currTimestamp = CurrentTimestamp() import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback assert(currTimestamp.isInstanceOf[CodegenFallback], "CurrentTimestamp should be a CodegenFallback") assert(currTimestamp.nullable == false, "CurrentTimestamp should not be nullable") import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext, ExprCode} val ctx = new CodegenContext // doGenCode is used when Expression.genCode is executed val ExprCode(code, _, _) = currTimestamp.genCode(ctx) scala> println(code) Object obj_0 = ((Expression) references[0]).eval(null); long value_0 = (Long) obj_0; |
Generating Java Source Code (ExprCode) For Code-Generated Expression Evaluation — doGenCode Method
|
1 2 3 4 5 |
doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode |
|
Note
|
doGenCode is part of Expression Contract to generate a Java source code (ExprCode) for code-generated expression evaluation.
|
doGenCode requests the input CodegenContext to add itself to the references.
doGenCode walks down the expression tree to find Nondeterministic expressions and for every Nondeterministic expression does the following:
-
Requests the input
CodegenContextto add it to the references -
Requests the input
CodegenContextto addPartitionInitializationStatement that is a Java code block as follows:123456((Nondeterministic) references[[childIndex]]).initialize(partitionIndex);
In the end, doGenCode generates a plain Java source code block that is one of the following code blocks per the nullable flag. doGenCode copies the input ExprCode with the code block added (as the code property).
|
1 2 3 4 5 6 7 8 9 10 11 12 |
doGenCode Code Block for nullable flag enabled [placeHolder] Object [objectTerm] = ((Expression) references[[idx]]).eval([input]); boolean [isNull] = [objectTerm] == null; [javaType] [value] = [defaultValue]; if (![isNull]) { [value] = ([boxedType]) [objectTerm]; } |
|
1 2 3 4 5 6 7 8 |
doGenCode Code Block for nullable flag disabled [placeHolder] Object [objectTerm] = ((Expression) references[[idx]]).eval([input]); [javaType] [value] = ([boxedType]) [objectTerm]; |
