spark技术分享GenerateUnsafeProjection
GenerateUnsafeProjection is a CodeGenerator that generates the bytecode for a UnsafeProjection for given expressions (i.e. CodeGenerator[Seq[Expression], UnsafeProjection]).
|
1 2 3 4 5 |
GenerateUnsafeProjection: Seq[Expression] => UnsafeProjection |
|
Tip
|
Enable Add the following line to
Refer to Logging. |
Generating UnsafeProjection — generate Method
|
1 2 3 4 5 6 7 |
generate( expressions: Seq[Expression], subexpressionEliminationEnabled: Boolean): UnsafeProjection |
generate canonicalize the input expressions followed by generating a JVM bytecode for a UnsafeProjection for the expressions.
|
Note
|
|
canonicalize Method
|
1 2 3 4 5 |
canonicalize(in: Seq[Expression]): Seq[Expression] |
canonicalize removes unnecessary Alias expressions.
Internally, canonicalize uses ExpressionCanonicalizer rule executor (that in turn uses just one CleanExpressions expression rule).
Generating JVM Bytecode For UnsafeProjection For Given Expressions (With Optional Subexpression Elimination) — create Method
|
1 2 3 4 5 6 7 8 |
create( expressions: Seq[Expression], subexpressionEliminationEnabled: Boolean): UnsafeProjection create(references: Seq[Expression]): UnsafeProjection (1) |
-
Calls the former
createwithsubexpressionEliminationEnabledflag off
create first creates a CodegenContext and an Java source code for the input expressions.
create creates a code body with public java.lang.Object generate(Object[] references) method that creates a SpecificUnsafeProjection.
|
1 2 3 4 5 6 7 8 9 10 11 |
public java.lang.Object generate(Object[] references) { return new SpecificUnsafeProjection(references); } class SpecificUnsafeProjection extends UnsafeProjection { ... } |
create creates a CodeAndComment with the code body and comment placeholders.
You should see the following DEBUG message in the logs:
|
1 2 3 4 5 6 |
DEBUG GenerateUnsafeProjection: code for [expressions]: [code] |
|
Tip
|
Enable
See CodeGenerator. |
create requests CodeGenerator to compile the Java source code to JVM bytecode (using Janino).
create requests CodegenContext for references and requests the compiled class to create a SpecificUnsafeProjection for the input references that in the end is the final UnsafeProjection.
|
Note
|
(Single-argument) create is part of CodeGenerator Contract.
|
Creating ExprCode for Expressions (With Optional Subexpression Elimination) — createCode Method
|
1 2 3 4 5 6 7 8 |
createCode( ctx: CodegenContext, expressions: Seq[Expression], useSubexprElimination: Boolean = false): ExprCode |
createCode requests the input CodegenContext to generate a Java source code for code-generated evaluation of every expression in the input expressions.
createCode…FIXME
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
import org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext val ctx = new CodegenContext // Use Catalyst DSL import org.apache.spark.sql.catalyst.dsl.expressions._ val expressions = "hello".expr.as("world") :: "hello".expr.as("world") :: Nil import org.apache.spark.sql.catalyst.expressions.codegen.GenerateUnsafeProjection val eval = GenerateUnsafeProjection.createCode(ctx, expressions, useSubexprElimination = true) scala> println(eval.code) mutableStateArray1[0].reset(); mutableStateArray2[0].write(0, ((UTF8String) references[0] /* literal */)); mutableStateArray2[0].write(1, ((UTF8String) references[1] /* literal */)); mutableStateArray[0].setTotalSize(mutableStateArray1[0].totalSize()); scala> println(eval.value) mutableStateArray[0] |
|
Note
|
|
