spark-sql-spark技术分享-第31页

UnresolvedGenerator

2012-10-05admin阅读(1305)

UnresolvedGenerator Expression

UnresolvedGenerator is a Generator that represents an unresolved generator in a logical query plan.

UnresolvedGenerator is created exclusively when AstBuilder is requested to withGenerate (as part of Generate logical operator) for SQL’s LATERAL VIEW (in SELECT or FROM clauses).



// SQLs borrowed from https://cwiki.apache.org/confluence/display/Hive/LanguageManual+LateralView
val sqlText = """
  SELECT pageid, adid
  FROM pageAds LATERAL VIEW explode(adid_list) adTable AS adid
"""
// Register pageAds table
Seq(
  ("front_page", Array(1, 2, 3)),
  ("contact_page", Array(3, 4, 5)))
  .toDF("pageid", "adid_list")
  .createOrReplaceTempView("pageAds")
val query = sql(sqlText)
val logicalPlan = query.queryExecution.logical
scala> println(logicalPlan.numberedTreeString)
00 'Project ['pageid, 'adid]
01 +- 'Generate 'explode('adid_list), false, adtable, ['adid]
02    +- 'UnresolvedRelation `pageAds`

import org.apache.spark.sql.catalyst.plans.logical.Generate
val generator = logicalPlan.collectFirst { case g: Generate => g.generator }.get

scala> :type generator
org.apache.spark.sql.catalyst.expressions.Generator

import org.apache.spark.sql.catalyst.analysis.UnresolvedGenerator
scala> generator.isInstanceOf[UnresolvedGenerator]
res1: Boolean = true

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

// SQLs borrowed from https://cwiki.apache.org/confluence/display/Hive/LanguageManual+LateralView

val sqlText = """

SELECT pageid, adid

FROM pageAds LATERAL VIEW explode(adid_list) adTable AS adid

"""

// Register pageAds table

Seq(

("front_page", Array(1, 2, 3)),

("contact_page", Array(3, 4, 5)))

.toDF("pageid", "adid_list")

.createOrReplaceTempView("pageAds")

val query = sql(sqlText)

val logicalPlan = query.queryExecution.logical

scala> println(logicalPlan.numberedTreeString)

00 'Project ['pageid, 'adid]

01 +- 'Generate 'explode('adid_list), false, adtable, ['adid]

02 +- 'UnresolvedRelation `pageAds`

import org.apache.spark.sql.catalyst.plans.logical.Generate

val generator = logicalPlan.collectFirst { case g: Generate => g.generator }.get

scala> :type generator

org.apache.spark.sql.catalyst.expressions.Generator

import org.apache.spark.sql.catalyst.analysis.UnresolvedGenerator

scala> generator.isInstanceOf[UnresolvedGenerator]

res1: Boolean = true

UnresolvedGenerator can never be resolved (and is replaced at analysis phase).

Given UnresolvedGenerator can never be resolved it should not come as a surprise that it cannot be evaluated either (i.e. produce a value given an internal row). When requested to evaluate, UnresolvedGenerator simply reports a UnsupportedOperationException.



Cannot evaluate expression: [this]

1

2

3

4

5

Cannot evaluate expression: [this]

Note	`UnresolvedGenerator` is resolved to a concrete Generator expression when ResolveFunctions logical resolution rule is executed.

Note	`UnresolvedGenerator` is similar to UnresolvedFunction and differs mostly by the type (to make Spark development with Scala easier?)

Creating UnresolvedGenerator Instance

UnresolvedGenerator takes the following when created:

FunctionIdentifier
Child expressions

UnresolvedFunction

2012-10-04admin阅读(1704)

UnresolvedFunction Unevaluable Expression — Logical Representation of Functions in Queries

UnresolvedFunction is an Catalyst expression that represents a function (application) in a logical query plan.

UnresolvedFunction is created as a result of the following:

callUDF standard function
RelationalGroupedDataset.agg operator with aggregation functions specified by name (that converts function names to UnresolvedFunction expressions)
AstBuilder is requested to visitFunctionCall (in SQL queries)

UnresolvedFunction can never be resolved (and is replaced at analysis phase).

Note	`UnresolvedFunction` is first looked up in LookupFunctions logical rule and then resolved in ResolveFunctions logical resolution rule.

Given UnresolvedFunction can never be resolved it should not come as a surprise that it cannot be evaluated either (i.e. produce a value given an internal row). When requested to evaluate, UnresolvedFunction simply reports a UnsupportedOperationException.



Cannot evaluate expression: [this]

1

2

3

4

5

Cannot evaluate expression: [this]

Note	Unevaluable expressions are expressions that have to be replaced by some other expressions during analysis or optimization (or they fail analysis).

Tip	Use Catalyst DSL’s function or distinctFunction to create a `UnresolvedFunction` with isDistinct flag off and on, respectively.



// Using Catalyst DSL to create UnresolvedFunctions
import org.apache.spark.sql.catalyst.dsl.expressions._

// Scala Symbols supported only
val f = 'f.function()
scala> :type f
org.apache.spark.sql.catalyst.analysis.UnresolvedFunction

scala> f.isDistinct
res0: Boolean = false

val g = 'g.distinctFunction()
scala> g.isDistinct
res1: Boolean = true

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

// Using Catalyst DSL to create UnresolvedFunctions

import org.apache.spark.sql.catalyst.dsl.expressions._

// Scala Symbols supported only

val f = 'f.function()

scala> :type f

org.apache.spark.sql.catalyst.analysis.UnresolvedFunction

scala> f.isDistinct

res0: Boolean = false

val g = 'g.distinctFunction()

scala> g.isDistinct

res1: Boolean = true

Creating UnresolvedFunction (With Database Undefined) — `apply` Factory Method



apply(name: String, children: Seq[Expression], isDistinct: Boolean): UnresolvedFunction

1

2

3

4

5

apply(name: String, children: Seq[Expression], isDistinct: Boolean): UnresolvedFunction

apply creates a FunctionIdentifier with the name and no database first and then creates a UnresolvedFunction with the FunctionIdentifier, children and isDistinct flag.

Note	`apply` is used when: callUDF standard function is used `RelationalGroupedDataset` is requested to agg with aggregation functions specified by name (and converts function names to UnresolvedFunction expressions)

Creating UnresolvedFunction Instance

UnresolvedFunction takes the following when created:

FunctionIdentifier
Child expressions
isDistinct flag

UnresolvedAttribute

2012-10-03admin阅读(2528)

UnresolvedAttribute Leaf Expression

UnresolvedAttribute is a named Attribute leaf expression (i.e. it has a name) that represents a reference to an entity in a logical query plan.

UnresolvedAttribute is created when:

AstBuilder is requested to visitDereference
LogicalPlan is requested to resolve an attribute by name parts
DescribeColumnCommand is executed

UnresolvedAttribute can never be resolved (and is replaced at analysis phase).

Note	`UnresolvedAttribute` is resolved when `Analyzer` is executed by the following logical resolution rules: ResolveReferences `ResolveMissingReferences` `ResolveDeserializer` `ResolveSubquery`

Given UnresolvedAttribute can never be resolved it should not come as a surprise that it cannot be evaluated either (i.e. produce a value given an internal row). When requested to evaluate, UnresolvedAttribute simply reports a UnsupportedOperationException.



Cannot evaluate expression: [this]

1

2

3

4

5

Cannot evaluate expression: [this]

UnresolvedAttribute takes name parts when created.



import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute

scala> val t1 = UnresolvedAttribute("t1")
t1: org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute = 't1

scala> val t2 = UnresolvedAttribute("db1.t2")
t2: org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute = 'db1.t2

scala> println(s"Number of name parts: ${t2.nameParts.length}")
Number of name parts: 2

scala> println(s"Name parts: ${t2.nameParts.mkString(",")}")
Name parts: db1,t2

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute

scala> val t1 = UnresolvedAttribute("t1")

t1: org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute = 't1

scala> val t2 = UnresolvedAttribute("db1.t2")

t2: org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute = 'db1.t2

scala> println(s"Number of name parts: ${t2.nameParts.length}")

Number of name parts: 2

scala> println(s"Name parts: ${t2.nameParts.mkString(",")}")

Name parts: db1,t2

UnresolvedAttribute can be created with a fully-qualified name with dots to separate name parts.



apply(name: String): UnresolvedAttribute

1

2

3

4

5

apply(name: String): UnresolvedAttribute

Tip

Use backticks () around names with dots (.) to disable them as separators.

The following is a two-part attribute name with a.b and c name parts.



`a.b`.c

1

2

3

4

5

`a.b`.c

UnresolvedAttribute can also be created without the dots with the special meaning.



import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute
val attr1 = UnresolvedAttribute.quoted("a.b.c")
scala> println(s"Number of name parts: ${attr1.nameParts.length}")
Number of name parts: 1

1

2

3

4

5

6

7

8

import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute

val attr1 = UnresolvedAttribute.quoted("a.b.c")

scala> println(s"Number of name parts: ${attr1.nameParts.length}")

Number of name parts: 1

Note

Catalyst DSL defines two Scala implicits to create an UnresolvedAttribute:

StringToAttributeConversionHelper is a Scala implicit class that converts $"colName" into an UnresolvedAttribute
symbolToUnresolvedAttribute is a Scala implicit method that converts 'colName into an UnresolvedAttribute

Both implicits are part of ExpressionConversions Scala trait of Catalyst DSL.

Import expressions object to get access to the expression conversions.



// Use `sbt console` with Spark libraries defined (in `build.sbt`)
import org.apache.spark.sql.catalyst.dsl.expressions._
scala> :type $"name"
org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute

import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute
val nameAttr: UnresolvedAttribute = 'name

scala> :type nameAttr
org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute

1

2

3

4

5

6

7

8

9

10

11

12

13

14

// Use `sbt console` with Spark libraries defined (in `build.sbt`)

import org.apache.spark.sql.catalyst.dsl.expressions._

scala> :type $"name"

org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute

import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute

val nameAttr: UnresolvedAttribute = 'name

scala> :type nameAttr

org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute

Note

A UnresolvedAttribute can be replaced by (resolved) a NamedExpression using an analyzed logical plan (of the structured query the attribute is part of).



val analyzedPlan = Seq((0, "zero")).toDF("id", "name").queryExecution.analyzed

import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute
val nameAttr = UnresolvedAttribute("name")
val nameResolved = analyzedPlan.resolveQuoted(
  name = nameAttr.name,
  resolver = spark.sessionState.analyzer.resolver).getOrElse(nameAttr)

scala> println(nameResolved.numberedTreeString)
00 name#47: string

scala> :type nameResolved
org.apache.spark.sql.catalyst.expressions.NamedExpression

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

val analyzedPlan = Seq((0, "zero")).toDF("id", "name").queryExecution.analyzed

import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute

val nameAttr = UnresolvedAttribute("name")

val nameResolved = analyzedPlan.resolveQuoted(

name = nameAttr.name,

resolver = spark.sessionState.analyzer.resolver).getOrElse(nameAttr)

scala> println(nameResolved.numberedTreeString)

00 name#47: string

scala> :type nameResolved

org.apache.spark.sql.catalyst.expressions.NamedExpression

UnixTimestamp

2012-10-02admin阅读(2253)

UnixTimestamp TimeZoneAware Binary Expression

UnixTimestamp is a binary expression with timezone support that represents unix_timestamp function (and indirectly to_date and to_timestamp).



import org.apache.spark.sql.functions.unix_timestamp
val c1 = unix_timestamp()

scala> c1.explain(true)
unix_timestamp(current_timestamp(), yyyy-MM-dd HH:mm:ss, None)

scala> println(c1.expr.numberedTreeString)
00 unix_timestamp(current_timestamp(), yyyy-MM-dd HH:mm:ss, None)
01 :- current_timestamp()
02 +- yyyy-MM-dd HH:mm:ss

import org.apache.spark.sql.catalyst.expressions.UnixTimestamp
scala> c1.expr.isInstanceOf[UnixTimestamp]
res0: Boolean = true

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

import org.apache.spark.sql.functions.unix_timestamp

val c1 = unix_timestamp()

scala> c1.explain(true)

unix_timestamp(current_timestamp(), yyyy-MM-dd HH:mm:ss, None)

scala> println(c1.expr.numberedTreeString)

00 unix_timestamp(current_timestamp(), yyyy-MM-dd HH:mm:ss, None)

01 :- current_timestamp()

02 +- yyyy-MM-dd HH:mm:ss

import org.apache.spark.sql.catalyst.expressions.UnixTimestamp

scala> c1.expr.isInstanceOf[UnixTimestamp]

res0: Boolean = true

Note	`UnixTimestamp` is `UnixTime` expression internally (as is `ToUnixTimestamp` expression).

UnixTimestamp supports StringType, DateType and TimestampType as input types for a time expression and returns LongType.



scala> c1.expr.eval()
res1: Any = 1493354303

1

2

3

4

5

6

scala> c1.expr.eval()

res1: Any = 1493354303

UnixTimestamp uses DateTimeUtils.newDateFormat for date/time format (as Java’s java.text.DateFormat).

UnaryExpression Contract

2012-10-01admin阅读(1912)

UnaryExpression Contract

UnaryExpression is…FIXME

`defineCodeGen` Method



defineCodeGen(
  ctx: CodegenContext,
  ev: ExprCode,
  f: String => String): ExprCode

1

2

3

4

5

6

7

8

defineCodeGen(

ctx: CodegenContext,

ev: ExprCode,

f: String => String): ExprCode

defineCodeGen…FIXME

Note	`defineCodeGen` is used when…FIXME

`nullSafeEval` Method



nullSafeEval(input: Any): Any

1

2

3

4

5

nullSafeEval(input: Any): Any

nullSafeEval simply fails with the following error (and is expected to be overrided to save null-check code):



UnaryExpressions must override either eval or nullSafeEval

1

2

3

4

5

UnaryExpressions must override either eval or nullSafeEval

Note	`nullSafeEval` is used exclusively when `UnaryExpression` is requested to eval.

Evaluating Expression — `eval` Method



eval(input: InternalRow): Any

1

2

3

4

5

eval(input: InternalRow): Any

Note	`eval` is part of Expression Contract for the interpreted (non-code-generated) expression evaluation, i.e. evaluating a Catalyst expression to a JVM object for a given internal binary row.

eval…FIXME

TypedImperativeAggregate

2012-09-30admin阅读(1973)

TypedImperativeAggregate — Contract for Imperative Aggregate Functions with Custom Aggregation Buffer

TypedImperativeAggregate is the contract for imperative aggregation functions that allows for an arbitrary user-defined java object to be used as internal aggregation buffer.

Table 1. TypedImperativeAggregate as ImperativeAggregate
ImperativeAggregate Method	Description
`aggBufferAttributes`
`aggBufferSchema`
initialize	Creates an aggregation buffer and puts it at mutableAggBufferOffset position in the input `buffer` InternalRow.
`inputAggBufferAttributes`

Table 2. TypedImperativeAggregate’s Direct Implementations
Name	Description
`ApproximatePercentile`
`Collect`
ComplexTypedAggregateExpression
`CountMinSketchAgg`
`HiveUDAFFunction`
`Percentile`

TypedImperativeAggregate Contract



package org.apache.spark.sql.catalyst.expressions.aggregate

abstract class TypedImperativeAggregate[T] extends ImperativeAggregate {
  def createAggregationBuffer(): T
  def deserialize(storageFormat: Array[Byte]): T
  def eval(buffer: T): Any
  def merge(buffer: T, input: T): T
  def serialize(buffer: T): Array[Byte]
  def update(buffer: T, input: InternalRow): T
}

1

2

3

4

5

6

7

8

9

10

11

12

13

14

package org.apache.spark.sql.catalyst.expressions.aggregate

abstract class TypedImperativeAggregate[T] extends ImperativeAggregate {

def createAggregationBuffer(): T

def deserialize(storageFormat: Array[Byte]): T

def eval(buffer: T): Any

def merge(buffer: T, input: T): T

def serialize(buffer: T): Array[Byte]

def update(buffer: T, input: InternalRow): T

}

Table 3. TypedImperativeAggregate Contract
Method	Description
`createAggregationBuffer`	Used exclusively when a `TypedImperativeAggregate` is initialized
`deserialize`
`eval`
`merge`
`serialize`
`update`

TypedAggregateExpression

2012-09-29admin阅读(1422)

TypedAggregateExpression Expression

TypedAggregateExpression is the contract for AggregateFunction expressions that…FIXME



package org.apache.spark.sql.execution.aggregate

trait TypedAggregateExpression extends AggregateFunction {
  // only required methods that have no implementation
  def aggregator: Aggregator[Any, Any, Any]
  def inputClass: Option[Class[_]]
  def inputDeserializer: Option[Expression]
  def inputSchema: Option[StructType]
  def withInputInfo(deser: Expression, cls: Class[_], schema: StructType): TypedAggregateExpression
}

1

2

3

4

5

6

7

8

9

10

11

12

13

14

package org.apache.spark.sql.execution.aggregate

trait TypedAggregateExpression extends AggregateFunction {

// only required methods that have no implementation

def aggregator: Aggregator[Any, Any, Any]

def inputClass: Option[Class[_]]

def inputDeserializer: Option[Expression]

def inputSchema: Option[StructType]

def withInputInfo(deser: Expression, cls: Class[_], schema: StructType): TypedAggregateExpression

}

TypedAggregateExpression is used when:

TypedColumn is requested to withInputType (for Dataset.select, KeyValueGroupedDataset.agg and RelationalGroupedDataset.agg operators)
Column is requested to generateAlias and named (for Dataset.select and KeyValueGroupedDataset.agg operators)
RelationalGroupedDataset is requested to alias (when RelationalGroupedDataset is requested to create a DataFrame from aggregate expressions)

Table 1. TypedAggregateExpression Contract
Method	Description
`aggregator`	Aggregator
`inputClass`	Used when…FIXME
`inputDeserializer`	Used when…FIXME
`inputSchema`	Used when…FIXME
`withInputInfo`	Used when…FIXME

Table 2. TypedAggregateExpressions
Aggregator	Description
ComplexTypedAggregateExpression
SimpleTypedAggregateExpression

Creating TypedAggregateExpression — `apply` Factory Method



apply[BUF : Encoder, OUT : Encoder](
  aggregator: Aggregator[_, BUF, OUT]): TypedAggregateExpression

1

2

3

4

5

6

apply[BUF : Encoder, OUT : Encoder](

aggregator: Aggregator[_, BUF, OUT]): TypedAggregateExpression

apply…FIXME

Note	`apply` is used exclusively when `Aggregator` is requested to convert itself to a TypedColumn.

SubqueryExpression

2012-09-28admin阅读(1788)

SubqueryExpression Contract — Expressions With Logical Query Plans

SubqueryExpression is the contract for expressions with logical query plans (i.e. PlanExpression[LogicalPlan]).



package org.apache.spark.sql.catalyst.expressions

abstract class SubqueryExpression(
    plan: LogicalPlan,
    children: Seq[Expression],
    exprId: ExprId) extends PlanExpression[LogicalPlan] {
  // only required methods that have no implementation
  // the others follow
  override def withNewPlan(plan: LogicalPlan): SubqueryExpression
}

1

2

3

4

5

6

7

8

9

10

11

12

13

14

package org.apache.spark.sql.catalyst.expressions

abstract class SubqueryExpression(

plan: LogicalPlan,

children: Seq[Expression],

exprId: ExprId) extends PlanExpression[LogicalPlan] {

// only required methods that have no implementation

// the others follow

override def withNewPlan(plan: LogicalPlan): SubqueryExpression

}

Table 1. (Subset of) SubqueryExpression Contract
Method	Description
`withNewPlan`	Used when: `CTESubstitution` substitution analyzer rule is requested to `substituteCTE` `ResolveReferences` logical resolution rule is requested to dedupRight and dedupOuterReferencesInSubquery `ResolveSubquery` logical resolution rule is requested to resolveSubQuery `UpdateOuterReferences` logical rule is executed `ResolveTimeZone` logical resolution rule is executed `SubqueryExpression` is requested for a canonicalized version `OptimizeSubqueries` logical query optimization is executed `CacheManager` is requested to replace logical query segments with cached query plans

Table 2. SubqueryExpressions
SubqueryExpression	Description
Exists
ListQuery
ScalarSubquery

SubqueryExpression is resolved when the children are resolved and the subquery logical plan is resolved.

references…FIXME

semanticEquals…FIXME

canonicalize…FIXME

`hasInOrExistsSubquery` Object Method



hasInOrExistsSubquery(e: Expression): Boolean

1

2

3

4

5

hasInOrExistsSubquery(e: Expression): Boolean

hasInOrExistsSubquery…FIXME

Note	`hasInOrExistsSubquery` is used when…FIXME

`hasCorrelatedSubquery` Object Method



hasCorrelatedSubquery(e: Expression): Boolean

1

2

3

4

5

hasCorrelatedSubquery(e: Expression): Boolean

hasCorrelatedSubquery…FIXME

Note	`hasCorrelatedSubquery` is used when…FIXME

`hasSubquery` Object Method



hasSubquery(e: Expression): Boolean

1

2

3

4

5

hasSubquery(e: Expression): Boolean

hasSubquery…FIXME

Note	`hasSubquery` is used when…FIXME

Creating SubqueryExpression Instance

SubqueryExpression takes the following when created:

Subquery logical plan
Child expressions
Expression ID (as ExprId)

StaticInvoke

2012-09-27admin阅读(1631)

StaticInvoke Non-SQL Expression

StaticInvoke is an expression with no SQL representation that represents a static method call in Scala or Java.

StaticInvoke supports Java code generation (aka whole-stage codegen) to evaluate itself.

StaticInvoke is created when:

ScalaReflection is requested for the deserializer or serializer for a Scala type
RowEncoder is requested for deserializerFor or serializer for a Scala type
JavaTypeInference is requested for deserializerFor or serializerFor



import org.apache.spark.sql.types.StructType
val schema = new StructType()
  .add($"id".long.copy(nullable = false))
  .add($"name".string.copy(nullable = false))

import org.apache.spark.sql.catalyst.encoders.RowEncoder
val encoder = RowEncoder(schema)
scala> println(encoder.serializer(0).numberedTreeString)
00 validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, id), LongType) AS id#1640L
01 +- validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, id), LongType)
02    +- getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, id)
03       +- assertnotnull(input[0, org.apache.spark.sql.Row, true])
04          +- input[0, org.apache.spark.sql.Row, true]

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

import org.apache.spark.sql.types.StructType

val schema = new StructType()

.add($"id".long.copy(nullable = false))

.add($"name".string.copy(nullable = false))

import org.apache.spark.sql.catalyst.encoders.RowEncoder

val encoder = RowEncoder(schema)

scala> println(encoder.serializer(0).numberedTreeString)

00 validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, id), LongType) AS id#1640L

01 +- validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, id), LongType)

02 +- getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, id)

03 +- assertnotnull(input[0, org.apache.spark.sql.Row, true])

04 +- input[0, org.apache.spark.sql.Row, true]

Note	`StaticInvoke` is similar to `CallMethodViaReflection` expression.

Creating StaticInvoke Instance

StaticInvoke takes the following when created:

Target object of the static call
Data type of the return value of the method
Name of the method to call on the static object
Optional expressions to pass as input arguments to the function
Flag to control whether to propagate nulls or not (enabled by default). If any of the arguments is null, null is returned instead of calling the function

Star

2012-09-26admin阅读(1606)

Star Expression Contract

Star is a contract of leaf and named expressions that…FIXME



package org.apache.spark.sql.catalyst.analysis

abstract class Star extends LeafExpression with NamedExpression {
  // only required methods that have no implementation
  def expand(input: LogicalPlan, resolver: Resolver): Seq[NamedExpression]
}

1

2

3

4

5

6

7

8

9

10

package org.apache.spark.sql.catalyst.analysis

abstract class Star extends LeafExpression with NamedExpression {

// only required methods that have no implementation

def expand(input: LogicalPlan, resolver: Resolver): Seq[NamedExpression]

}

Table 1. Star Contract
Method	Description
`expand`	Used exclusively when `ResolveReferences` logical resolution rule is requested to expand `Star` expressions in the following logical operators: ScriptTransformation Project and Aggregate

Table 2. Stars
Star	Description
ResolvedStar
UnresolvedRegex
UnresolvedStar

spark-sql 第31页

UnresolvedGenerator

UnresolvedGenerator Expression

Creating UnresolvedGenerator Instance

UnresolvedFunction

UnresolvedFunction Unevaluable Expression — Logical Representation of Functions in Queries

Creating UnresolvedFunction (With Database Undefined) — `apply` Factory Method

Creating UnresolvedFunction Instance

UnresolvedAttribute

UnresolvedAttribute Leaf Expression

UnixTimestamp

UnixTimestamp TimeZoneAware Binary Expression

UnaryExpression Contract

UnaryExpression Contract

`defineCodeGen` Method

`nullSafeEval` Method

Evaluating Expression — `eval` Method

TypedImperativeAggregate

TypedImperativeAggregate — Contract for Imperative Aggregate Functions with Custom Aggregation Buffer

TypedImperativeAggregate Contract

TypedAggregateExpression

TypedAggregateExpression Expression

Creating TypedAggregateExpression — `apply` Factory Method

SubqueryExpression

SubqueryExpression Contract — Expressions With Logical Query Plans

`hasInOrExistsSubquery` Object Method

`hasCorrelatedSubquery` Object Method

`hasSubquery` Object Method

Creating SubqueryExpression Instance

StaticInvoke

StaticInvoke Non-SQL Expression

Creating StaticInvoke Instance

Star

Star Expression Contract

欢迎关注：spark技术分享

关注公众号：spark技术分享

QQ咨询

回顶部

spark-sql 第31页

UnresolvedGenerator Expression

Creating UnresolvedGenerator Instance

UnresolvedFunction Unevaluable Expression — Logical Representation of Functions in Queries

Creating UnresolvedFunction (With Database Undefined) — apply Factory Method

Creating UnresolvedFunction Instance

UnresolvedAttribute Leaf Expression

UnixTimestamp TimeZoneAware Binary Expression

UnaryExpression Contract

defineCodeGen Method

nullSafeEval Method

Evaluating Expression — eval Method

TypedImperativeAggregate — Contract for Imperative Aggregate Functions with Custom Aggregation Buffer

TypedImperativeAggregate Contract

TypedAggregateExpression Expression

Creating TypedAggregateExpression — apply Factory Method

SubqueryExpression Contract — Expressions With Logical Query Plans

hasInOrExistsSubquery Object Method

hasCorrelatedSubquery Object Method

hasSubquery Object Method

Creating SubqueryExpression Instance

StaticInvoke Non-SQL Expression

Creating StaticInvoke Instance

Star Expression Contract

欢迎关注：spark技术分享

关注公众号：spark技术分享

QQ咨询

回顶部

Creating UnresolvedFunction (With Database Undefined) — `apply` Factory Method

`defineCodeGen` Method

`nullSafeEval` Method

Evaluating Expression — `eval` Method

Creating TypedAggregateExpression — `apply` Factory Method

`hasInOrExistsSubquery` Object Method

`hasCorrelatedSubquery` Object Method

`hasSubquery` Object Method