spark技术分享StaticInvoke Non-SQL Expression
StaticInvoke is an expression with no SQL representation that represents a static method call in Scala or Java.
StaticInvoke supports Java code generation (aka whole-stage codegen) to evaluate itself.
StaticInvoke is created when:
-
ScalaReflectionis requested for the deserializer or serializer for a Scala type -
RowEncoder is requested for
deserializerForor serializer for a Scala type -
JavaTypeInferenceis requested fordeserializerFororserializerFor
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
import org.apache.spark.sql.types.StructType val schema = new StructType() .add($"id".long.copy(nullable = false)) .add($"name".string.copy(nullable = false)) import org.apache.spark.sql.catalyst.encoders.RowEncoder val encoder = RowEncoder(schema) scala> println(encoder.serializer(0).numberedTreeString) 00 validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, id), LongType) AS id#1640L 01 +- validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, id), LongType) 02 +- getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, id) 03 +- assertnotnull(input[0, org.apache.spark.sql.Row, true]) 04 +- input[0, org.apache.spark.sql.Row, true] |
|
Note
|
StaticInvoke is similar to CallMethodViaReflection expression.
|
Creating StaticInvoke Instance
StaticInvoke takes the following when created:
-
Name of the method to call on the static object
-
Optional expressions to pass as input arguments to the function
-
Flag to control whether to propagate
nullsor not (enabled by default). If any of the arguments isnull,nullis returned instead of calling the function
