spark-sql-spark技术分享-第36页

CumeDist

2012-08-16admin阅读(1296)

CumeDist Declarative Window Aggregate Function Expression

CumeDist is a SizeBasedWindowFunction and a RowNumberLike expression that is used for the following:

cume_dist standard function (Dataset API)
cume_dist SQL function

CumeDist takes no input parameters when created.



import org.apache.spark.sql.catalyst.expressions.CumeDist
val cume_dist = CumeDist()

1

2

3

4

5

6

import org.apache.spark.sql.catalyst.expressions.CumeDist

val cume_dist = CumeDist()

CumeDist uses cume_dist for the user-facing name.



import org.apache.spark.sql.catalyst.expressions.CumeDist
val cume_dist = CumeDist()
scala> println(cume_dist)
cume_dist()

1

2

3

4

5

6

7

8

import org.apache.spark.sql.catalyst.expressions.CumeDist

val cume_dist = CumeDist()

scala> println(cume_dist)

cume_dist()

As an WindowFunction expression (indirectly), CumeDist requires the SpecifiedWindowFrame (with the RangeFrame frame type, the UnboundedPreceding lower and the CurrentRow upper frame boundaries) as the frame.

Note	The frame for `CumeDist` expression is range-based instead of row-based, because it has to return the same value for tie values in a window (equal values per `ORDER BY` specification).

As a DeclarativeAggregate expression (indirectly), CumeDist defines the evaluateExpression expression which returns the final value when CumeDist is evaluated. The value uses the formula rowNumber / n where rowNumber is the row number in a window frame (the number of values before and including the current row) divided by the number of rows in the window frame.



import org.apache.spark.sql.catalyst.expressions.CumeDist
val cume_dist = CumeDist()
scala> println(cume_dist.evaluateExpression.numberedTreeString)
00 (cast(rowNumber#0 as double) / cast(window__partition__size#1 as double))
01 :- cast(rowNumber#0 as double)
02 :  +- rowNumber#0: int
03 +- cast(window__partition__size#1 as double)
04    +- window__partition__size#1: int

1

2

3

4

5

6

7

8

9

10

11

12

import org.apache.spark.sql.catalyst.expressions.CumeDist

val cume_dist = CumeDist()

scala> println(cume_dist.evaluateExpression.numberedTreeString)

00 (cast(rowNumber#0 as double) / cast(window__partition__size#1 as double))

01 :- cast(rowNumber#0 as double)

02 : +- rowNumber#0: int

03 +- cast(window__partition__size#1 as double)

04 +- window__partition__size#1: int

CreateNamedStructUnsafe

2012-08-15admin阅读(1216)

CreateNamedStructUnsafe Expression

CreateNamedStructUnsafe is a CreateNamedStructLike expression that…FIXME

Generating Java Source Code (ExprCode) For Code-Generated Expression Evaluation — `doGenCode` Method



doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode

1

2

3

4

5

doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode

Note	`doGenCode` is part of Expression Contract to generate a Java source code (ExprCode) for code-generated expression evaluation.

doGenCode…FIXME

CreateNamedStructLike Contract

2012-08-14admin阅读(1339)

CreateNamedStructLike Contract

CreateNamedStructLike is the base of Catalyst expressions that FIXME.



package org.apache.spark.sql.catalyst.expressions

trait CreateNamedStructLike extends Expression {
  // no required properties (vals and methods) that have no implementation
}

1

2

3

4

5

6

7

8

9

package org.apache.spark.sql.catalyst.expressions

trait CreateNamedStructLike extends Expression {

// no required properties (vals and methods) that have no implementation

}

CreateNamedStructLike is not nullable.

CreateNamedStructLike is foldable only if all value expressions are.

Table 1. CreateNamedStructLikes (Direct Implementations)
CreateNamedStructLike	Description
CreateNamedStruct
CreateNamedStructUnsafe

Table 2. CreateNamedStructLike’s Internal Properties (e.g. Registries, Counters and Flags)
Name	Description
`dataType`
`nameExprs`	Catalyst expressions for names
`names`
`valExprs`	Catalyst expressions for values

Checking Input Data Types — `checkInputDataTypes` Method



checkInputDataTypes(): TypeCheckResult

1

2

3

4

5

checkInputDataTypes(): TypeCheckResult

Note	`checkInputDataTypes` is part of the Expression Contract to verify (check the correctness of) the input data types.

checkInputDataTypes…FIXME

Evaluating Expression — `eval` Method



eval(input: InternalRow): Any

1

2

3

4

5

eval(input: InternalRow): Any

Note	`eval` is part of Expression Contract for the interpreted (non-code-generated) expression evaluation, i.e. evaluating a Catalyst expression to a JVM object for a given internal binary row.

eval…FIXME

CreateNamedStruct

2012-08-13admin阅读(1415)

CreateNamedStruct Expression

CreateNamedStruct is a CreateNamedStructLike expression that…FIXME

CreateNamedStruct uses named_struct for the user-facing name.



// Using Catalyst DSL
import org.apache.spark.sql.catalyst.dsl.expressions._
val s = namedStruct("*")
scala> println(s)
named_struct(*)

1

2

3

4

5

6

7

8

9

// Using Catalyst DSL

import org.apache.spark.sql.catalyst.dsl.expressions._

val s = namedStruct("*")

scala> println(s)

named_struct(*)

CreateNamedStruct is registered in FunctionRegistry under the name of named_struct SQL function.



import org.apache.spark.sql.catalyst.FunctionIdentifier
val fid = FunctionIdentifier(funcName = "named_struct")
val className = spark.sessionState.functionRegistry.lookupFunction(fid).get.getClassName
scala> println(className)
org.apache.spark.sql.catalyst.expressions.CreateNamedStruct

val q = sql("SELECT named_struct('id', 0)")
// analyzed so the function is resolved already (using FunctionRegistry)
val analyzedPlan = q.queryExecution.analyzed
scala> println(analyzedPlan.numberedTreeString)
00 Project [named_struct(id, 0) AS named_struct(id, 0)#7]
01 +- OneRowRelation

val e = analyzedPlan.expressions.head.children.head
import org.apache.spark.sql.catalyst.expressions.CreateNamedStruct
assert(e.isInstanceOf[CreateNamedStruct])

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

import org.apache.spark.sql.catalyst.FunctionIdentifier

val fid = FunctionIdentifier(funcName = "named_struct")

val className = spark.sessionState.functionRegistry.lookupFunction(fid).get.getClassName

scala> println(className)

org.apache.spark.sql.catalyst.expressions.CreateNamedStruct

val q = sql("SELECT named_struct('id', 0)")

// analyzed so the function is resolved already (using FunctionRegistry)

val analyzedPlan = q.queryExecution.analyzed

scala> println(analyzedPlan.numberedTreeString)

00 Project [named_struct(id, 0) AS named_struct(id, 0)#7]

01 +- OneRowRelation

val e = analyzedPlan.expressions.head.children.head

import org.apache.spark.sql.catalyst.expressions.CreateNamedStruct

assert(e.isInstanceOf[CreateNamedStruct])

CreateNamedStruct is created when:

ScalaReflection, RowEncoder and JavaTypeInference are requested for a serializer of a type
TimeWindowing and ResolveCreateNamedStruct logical resolution rules are executed
CreateStruct is requested to create a CreateNamedStruct expression

CreateNamedStruct takes a collection of Catalyst expressions when created.

CreateNamedStruct generates Java source code (as ExprCode) for code-generated expression evaluation.



// You could also use Seq("*")
import org.apache.spark.sql.functions.lit
val exprs = Seq("a", 1).map(lit).map(_.expr)

import org.apache.spark.sql.catalyst.expressions.CreateNamedStruct
val createNamedStruct = CreateNamedStruct(exprs)

import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext, ExprCode}
val ctx = new CodegenContext
// doGenCode is used when Expression.genCode is executed
val ExprCode(code, _, _) = createNamedStruct.genCode(ctx)

// Helper methods
def trim(code: String): String = {
  code.trim.split("\n").map(_.trim).filter(line => line.nonEmpty).mkString("\n")
}
def prettyPrint(code: String) = println(trim(code))
// END: Helper methods

scala> println(trim(code))
Object[] values_0 = new Object[1];
if (false) {
values_0[0] = null;
} else {
values_0[0] = 1;
}
final InternalRow value_0 = new org.apache.spark.sql.catalyst.expressions.GenericInternalRow(values_0);
values_0 = null;

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

// You could also use Seq("*")

import org.apache.spark.sql.functions.lit

val exprs = Seq("a", 1).map(lit).map(_.expr)

import org.apache.spark.sql.catalyst.expressions.CreateNamedStruct

val createNamedStruct = CreateNamedStruct(exprs)

import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext, ExprCode}

val ctx = new CodegenContext

// doGenCode is used when Expression.genCode is executed

val ExprCode(code, _, _) = createNamedStruct.genCode(ctx)

// Helper methods

def trim(code: String): String = {

code.trim.split("\n").map(_.trim).filter(line => line.nonEmpty).mkString("\n")

}

def prettyPrint(code: String) = println(trim(code))

// END: Helper methods

scala> println(trim(code))

Object[] values_0 = new Object[1];

if (false) {

values_0[0] = null;

} else {

values_0[0] = 1;

}

final InternalRow value_0 = new org.apache.spark.sql.catalyst.expressions.GenericInternalRow(values_0);

values_0 = null;

Tip

Use namedStruct operator from Catalyst DSL’s expressions to create a CreateNamedStruct expression.



import org.apache.spark.sql.catalyst.dsl.expressions._
val s = namedStruct()
scala> :type s
org.apache.spark.sql.catalyst.expressions.Expression

import org.apache.spark.sql.catalyst.expressions.CreateNamedStruct
assert(s.isInstanceOf[CreateNamedStruct])

val s = namedStruct("*")
scala> println(s)
named_struct(*)

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

import org.apache.spark.sql.catalyst.dsl.expressions._

val s = namedStruct()

scala> :type s

org.apache.spark.sql.catalyst.expressions.Expression

import org.apache.spark.sql.catalyst.expressions.CreateNamedStruct

assert(s.isInstanceOf[CreateNamedStruct])

val s = namedStruct("*")

scala> println(s)

named_struct(*)

Generating Java Source Code (ExprCode) For Code-Generated Expression Evaluation — `doGenCode` Method



doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode

1

2

3

4

5

doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode

Note	`doGenCode` is part of Expression Contract to generate a Java source code (ExprCode) for code-generated expression evaluation.

doGenCode…FIXME

CreateArray

2012-08-12admin阅读(1086)

CreateArray

CreateArray is…FIXME

ComplexTypedAggregateExpression

2012-08-11admin阅读(1270)

ComplexTypedAggregateExpression

ComplexTypedAggregateExpression is…FIXME

ComplexTypedAggregateExpression is created when…FIXME

Creating ComplexTypedAggregateExpression Instance

ComplexTypedAggregateExpression takes the following when created:

Aggregator
Optional input deserializer expression
Optional Java class for the input
Optional schema for the input
Buffer serializer (as a collection of named expressions)
Buffer deserializer expression
Output serializer (as a collection of expressions)
DataType
nullable flag
mutableAggBufferOffset (default: 0)
inputAggBufferOffset (default: 0)

CollectionGenerator

2012-08-10admin阅读(1296)

CollectionGenerator Generator Expression Contract

CollectionGenerator is the contract in Spark SQL for Generator expressions that generate a collection object (i.e. an array or map) and (at execution time) use a different path for whole-stage Java code generation (while executing GenerateExec physical operator with Whole-Stage Java Code Generation enabled).



package org.apache.spark.sql.catalyst.expressions

trait CollectionGenerator extends Generator {
  def collectionType: DataType = dataType
  def inline: Boolean
  def position: Boolean
}

1

2

3

4

5

6

7

8

9

10

11

package org.apache.spark.sql.catalyst.expressions

trait CollectionGenerator extends Generator {

def collectionType: DataType = dataType

def inline: Boolean

def position: Boolean

}

Table 1. CollectionGenerator Contract
Method	Description
`collectionType`	The type of the returned collection object. Used when…
`inline`	Flag whether to inline rows during whole-stage Java code generation. Used when…
`position`	Flag whether to include the positions of elements within the result collection. Used when…

Table 2. CollectionGenerators
CollectionGenerator	Description
Inline
ExplodeBase
Explode
PosExplode

CodegenFallback

2012-08-09admin阅读(1667)

CodegenFallback Contract — Catalyst Expressions with Fallback Code Generation Mode

CodegenFallback is the contract of Catalyst expressions that do not support a Java code generation and want to fall back to interpreted mode (aka fallback mode).

CodegenFallback is used when CollapseCodegenStages physical optimization is requested to execute (and enforce whole-stage codegen requirements for Catalyst expressions).



package org.apache.spark.sql.catalyst.expressions.codegen

trait CodegenFallback extends Expression {
  // No properties (vals and methods) that have no implementation
}

1

2

3

4

5

6

7

8

9

package org.apache.spark.sql.catalyst.expressions.codegen

trait CodegenFallback extends Expression {

// No properties (vals and methods) that have no implementation

}

Table 1. (Some Examples of) CodegenFallbacks
CodegenFallback	Description
`CurrentDate`
`CurrentTimestamp`
`Cube`
JsonToStructs
`Rollup`
`StructsToJson`


Example — CurrentTimestamp Expression with nullable flag disabled

import org.apache.spark.sql.catalyst.expressions.CurrentTimestamp
val currTimestamp = CurrentTimestamp()

import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
assert(currTimestamp.isInstanceOf[CodegenFallback], "CurrentTimestamp should be a CodegenFallback")

assert(currTimestamp.nullable == false, "CurrentTimestamp should not be nullable")

import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext, ExprCode}
val ctx = new CodegenContext
// doGenCode is used when Expression.genCode is executed
val ExprCode(code, _, _) = currTimestamp.genCode(ctx)

scala> println(code)

Object obj_0 = ((Expression) references[0]).eval(null);
        long value_0 = (Long) obj_0;

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

Example — CurrentTimestamp Expression with nullable flag disabled

import org.apache.spark.sql.catalyst.expressions.CurrentTimestamp

val currTimestamp = CurrentTimestamp()

import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback

assert(currTimestamp.isInstanceOf[CodegenFallback], "CurrentTimestamp should be a CodegenFallback")

assert(currTimestamp.nullable == false, "CurrentTimestamp should not be nullable")

import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext, ExprCode}

val ctx = new CodegenContext

// doGenCode is used when Expression.genCode is executed

val ExprCode(code, _, _) = currTimestamp.genCode(ctx)

scala> println(code)

Object obj_0 = ((Expression) references[0]).eval(null);

long value_0 = (Long) obj_0;

Generating Java Source Code (ExprCode) For Code-Generated Expression Evaluation — `doGenCode` Method



doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode

1

2

3

4

5

doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode

Note	`doGenCode` is part of Expression Contract to generate a Java source code (ExprCode) for code-generated expression evaluation.

doGenCode requests the input CodegenContext to add itself to the references.

doGenCode walks down the expression tree to find Nondeterministic expressions and for every Nondeterministic expression does the following:

Requests the input CodegenContext to add it to the references
Requests the input CodegenContext to addPartitionInitializationStatement that is a Java code block as follows:

((Nondeterministic) references[[childIndex]]) .initialize(partitionIndex);

1
2
3
4
5
6

((Nondeterministic) references[[childIndex]])
.initialize(partitionIndex);

In the end, doGenCode generates a plain Java source code block that is one of the following code blocks per the nullable flag. doGenCode copies the input ExprCode with the code block added (as the code property).


doGenCode Code Block for nullable flag enabled

[placeHolder]
Object [objectTerm] = ((Expression) references[[idx]]).eval([input]);
boolean [isNull] = [objectTerm] == null;
[javaType] [value] = [defaultValue];
if (![isNull]) {
  [value] = ([boxedType]) [objectTerm];
}

1

2

3

4

5

6

7

8

9

10

11

12

doGenCode Code Block for nullable flag enabled

[placeHolder]

Object [objectTerm] = ((Expression) references[[idx]]).eval([input]);

boolean [isNull] = [objectTerm] == null;

[javaType] [value] = [defaultValue];

if (![isNull]) {

[value] = ([boxedType]) [objectTerm];

}


doGenCode Code Block for nullable flag disabled

[placeHolder]
Object [objectTerm] = ((Expression) references[[idx]]).eval([input]);
[javaType] [value] = ([boxedType]) [objectTerm];

1

2

3

4

5

6

7

8

doGenCode Code Block for nullable flag disabled

[placeHolder]

Object [objectTerm] = ((Expression) references[[idx]]).eval([input]);

[javaType] [value] = ([boxedType]) [objectTerm];

Coalesce

2012-08-08admin阅读(1581)

Coalesce Expression

Coalesce is a Catalyst expression to represent coalesce standard function or SQL’s coalesce function in structured queries.

When created, Coalesce takes Catalyst expressions (as the children).



import org.apache.spark.sql.catalyst.expressions.Coalesce

// Use Catalyst DSL
import org.apache.spark.sql.catalyst.dsl.expressions._

import org.apache.spark.sql.functions.lit
val coalesceExpr = Coalesce(children = Seq(lit(null).expr % 1, lit(null).expr, 1d))
scala> println(coalesceExpr.numberedTreeString)
00 coalesce((null % 1), null, 1.0)
01 :- (null % 1)
02 :  :- null
03 :  +- 1
04 :- null
05 +- 1.0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

import org.apache.spark.sql.catalyst.expressions.Coalesce

// Use Catalyst DSL

import org.apache.spark.sql.catalyst.dsl.expressions._

import org.apache.spark.sql.functions.lit

val coalesceExpr = Coalesce(children = Seq(lit(null).expr % 1, lit(null).expr, 1d))

scala> println(coalesceExpr.numberedTreeString)

00 coalesce((null % 1), null, 1.0)

01 :- (null % 1)

02 : :- null

03 : +- 1

04 :- null

05 +- 1.0

Caution

FIXME Describe FunctionArgumentConversion and Coalesce

Spark Optimizer uses NullPropagation logical optimization to remove null literals (in the children expressions). That could result in a static evaluation that gives null value if all children expressions are null literals.



// FIXME
// Demo Coalesce with nulls only
// Demo Coalesce with null and non-null expressions that are optimized to one expression (in NullPropagation)
// Demo Coalesce with non-null expressions after NullPropagation optimization

1

2

3

4

5

6

7

8

// FIXME

// Demo Coalesce with nulls only

// Demo Coalesce with null and non-null expressions that are optimized to one expression (in NullPropagation)

// Demo Coalesce with non-null expressions after NullPropagation optimization

Coalesce is also created when:

Analyzer is requested to commonNaturalJoinProcessing for FullOuter join type
RewriteDistinctAggregates logical optimization is requested to rewrite
ExtractEquiJoinKeys Scala extractor is requested to destructure a logical plan
ColumnStat is requested to statExprs
IfNull expression is created
Nvl expression is created
Whenever Cast expression is used in Catalyst expressions (e.g. Average, Sum)

CallMethodViaReflection

2012-08-07admin阅读(924)

CallMethodViaReflection Expression

CallMethodViaReflection is an expression that represents a static method call in Scala or Java using reflect and java_method functions.

Note	`reflect` and `java_method` functions are only supported in SQL and expression modes.

Table 1. CallMethodViaReflection’s DataType to JVM Types Mapping
DataType	JVM Type
`BooleanType`	`java.lang.Boolean` / `scala.Boolean`
`ByteType`	`java.lang.Byte` / `Byte`
`ShortType`	`java.lang.Short` / `Short`
`IntegerType`	`java.lang.Integer` / `Int`
`LongType`	`java.lang.Long` / `Long`
`FloatType`	`java.lang.Float` / `Float`
`DoubleType`	`java.lang.Double` / `Double`
`StringType`	`String`



import org.apache.spark.sql.catalyst.expressions.CallMethodViaReflection
import org.apache.spark.sql.catalyst.expressions.Literal
scala> val expr = CallMethodViaReflection(
     |   Literal("java.time.LocalDateTime") ::
     |   Literal("now") :: Nil)
expr: org.apache.spark.sql.catalyst.expressions.CallMethodViaReflection = reflect(java.time.LocalDateTime, now)
scala> println(expr.numberedTreeString)
00 reflect(java.time.LocalDateTime, now)
01 :- java.time.LocalDateTime
02 +- now

// CallMethodViaReflection as the expression for reflect SQL function
val q = """
  select reflect("java.time.LocalDateTime", "now") as now
  """
val plan = spark.sql(q).queryExecution.logical
// CallMethodViaReflection shows itself under "reflect" name
scala> println(plan.numberedTreeString)
00 Project [reflect(java.time.LocalDateTime, now) AS now#39]
01 +- OneRowRelation$

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

import org.apache.spark.sql.catalyst.expressions.CallMethodViaReflection

import org.apache.spark.sql.catalyst.expressions.Literal

scala> val expr = CallMethodViaReflection(

| Literal("java.time.LocalDateTime") ::

| Literal("now") :: Nil)

expr: org.apache.spark.sql.catalyst.expressions.CallMethodViaReflection = reflect(java.time.LocalDateTime, now)

scala> println(expr.numberedTreeString)

00 reflect(java.time.LocalDateTime, now)

01 :- java.time.LocalDateTime

02 +- now

// CallMethodViaReflection as the expression for reflect SQL function

val q = """

select reflect("java.time.LocalDateTime", "now") as now

"""

val plan = spark.sql(q).queryExecution.logical

// CallMethodViaReflection shows itself under "reflect" name

scala> println(plan.numberedTreeString)

00 Project [reflect(java.time.LocalDateTime, now) AS now#39]

01 +- OneRowRelation$

CallMethodViaReflection supports a fallback mode for expression code generation.

Table 2. CallMethodViaReflection’s Properties
Property	Description
`dataType`	`StringType`
`deterministic`	Disabled (i.e. `false`)
`nullable`	Enabled (i.e. `true`)
`prettyName`	`reflect`

Note	`CallMethodViaReflection` is very similar to StaticInvoke expression.

spark-sql 第36页

CumeDist

CumeDist Declarative Window Aggregate Function Expression

CreateNamedStructUnsafe

CreateNamedStructUnsafe Expression

Generating Java Source Code (ExprCode) For Code-Generated Expression Evaluation — `doGenCode` Method

CreateNamedStructLike Contract

CreateNamedStructLike Contract

Checking Input Data Types — `checkInputDataTypes` Method

Evaluating Expression — `eval` Method

CreateNamedStruct

CreateNamedStruct Expression

Generating Java Source Code (ExprCode) For Code-Generated Expression Evaluation — `doGenCode` Method

CreateArray

CreateArray

ComplexTypedAggregateExpression

ComplexTypedAggregateExpression

Creating ComplexTypedAggregateExpression Instance

CollectionGenerator

CollectionGenerator Generator Expression Contract

CodegenFallback

CodegenFallback Contract — Catalyst Expressions with Fallback Code Generation Mode

Generating Java Source Code (ExprCode) For Code-Generated Expression Evaluation — `doGenCode` Method

Coalesce

Coalesce Expression

CallMethodViaReflection

CallMethodViaReflection Expression

欢迎关注：spark技术分享

关注公众号：spark技术分享

QQ咨询

回顶部

spark-sql 第36页

CumeDist Declarative Window Aggregate Function Expression

CreateNamedStructUnsafe Expression

Generating Java Source Code (ExprCode) For Code-Generated Expression Evaluation — doGenCode Method

CreateNamedStructLike Contract

Checking Input Data Types — checkInputDataTypes Method

Evaluating Expression — eval Method

CreateNamedStruct Expression

Generating Java Source Code (ExprCode) For Code-Generated Expression Evaluation — doGenCode Method

CreateArray

ComplexTypedAggregateExpression

Creating ComplexTypedAggregateExpression Instance

CollectionGenerator Generator Expression Contract

CodegenFallback Contract — Catalyst Expressions with Fallback Code Generation Mode

Generating Java Source Code (ExprCode) For Code-Generated Expression Evaluation — doGenCode Method

Coalesce Expression

CallMethodViaReflection Expression

欢迎关注：spark技术分享

关注公众号：spark技术分享

QQ咨询

回顶部

Generating Java Source Code (ExprCode) For Code-Generated Expression Evaluation — `doGenCode` Method

Checking Input Data Types — `checkInputDataTypes` Method

Evaluating Expression — `eval` Method

Generating Java Source Code (ExprCode) For Code-Generated Expression Evaluation — `doGenCode` Method

Generating Java Source Code (ExprCode) For Code-Generated Expression Evaluation — `doGenCode` Method