spark技术分享CreateNamedStruct Expression
CreateNamedStruct is a CreateNamedStructLike expression that…FIXME
CreateNamedStruct uses named_struct for the user-facing name.
|
1 2 3 4 5 6 7 8 9 |
// Using Catalyst DSL import org.apache.spark.sql.catalyst.dsl.expressions._ val s = namedStruct("*") scala> println(s) named_struct(*) |
CreateNamedStruct is registered in FunctionRegistry under the name of named_struct SQL function.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
import org.apache.spark.sql.catalyst.FunctionIdentifier val fid = FunctionIdentifier(funcName = "named_struct") val className = spark.sessionState.functionRegistry.lookupFunction(fid).get.getClassName scala> println(className) org.apache.spark.sql.catalyst.expressions.CreateNamedStruct val q = sql("SELECT named_struct('id', 0)") // analyzed so the function is resolved already (using FunctionRegistry) val analyzedPlan = q.queryExecution.analyzed scala> println(analyzedPlan.numberedTreeString) 00 Project [named_struct(id, 0) AS named_struct(id, 0)#7] 01 +- OneRowRelation val e = analyzedPlan.expressions.head.children.head import org.apache.spark.sql.catalyst.expressions.CreateNamedStruct assert(e.isInstanceOf[CreateNamedStruct]) |
CreateNamedStruct is created when:
-
ScalaReflection, RowEncoder and
JavaTypeInferenceare requested for a serializer of a type -
TimeWindowing and ResolveCreateNamedStruct logical resolution rules are executed
-
CreateStructis requested to create a CreateNamedStruct expression
CreateNamedStruct takes a collection of Catalyst expressions when created.
CreateNamedStruct generates Java source code (as ExprCode) for code-generated expression evaluation.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
// You could also use Seq("*") import org.apache.spark.sql.functions.lit val exprs = Seq("a", 1).map(lit).map(_.expr) import org.apache.spark.sql.catalyst.expressions.CreateNamedStruct val createNamedStruct = CreateNamedStruct(exprs) import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext, ExprCode} val ctx = new CodegenContext // doGenCode is used when Expression.genCode is executed val ExprCode(code, _, _) = createNamedStruct.genCode(ctx) // Helper methods def trim(code: String): String = { code.trim.split("\n").map(_.trim).filter(line => line.nonEmpty).mkString("\n") } def prettyPrint(code: String) = println(trim(code)) // END: Helper methods scala> println(trim(code)) Object[] values_0 = new Object[1]; if (false) { values_0[0] = null; } else { values_0[0] = 1; } final InternalRow value_0 = new org.apache.spark.sql.catalyst.expressions.GenericInternalRow(values_0); values_0 = null; |
|
Tip
|
Use
|
Generating Java Source Code (ExprCode) For Code-Generated Expression Evaluation — doGenCode Method
|
1 2 3 4 5 |
doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode |
|
Note
|
doGenCode is part of Expression Contract to generate a Java source code (ExprCode) for code-generated expression evaluation.
|
doGenCode…FIXME
