spark技术分享ScalarSubquery (ExecSubqueryExpression) Expression
ScalarSubquery is an ExecSubqueryExpression that can give exactly one value (i.e. the value of executing SubqueryExec subquery that can result in a single row and a single column or null if no row were computed).
|
Important
|
Spark SQL uses the name of ScalarSubquery twice to represent an ExecSubqueryExpression (this page) and a SubqueryExpression. It is confusing and you should not be anymore.
|
ScalarSubquery is created exclusively when PlanSubqueries physical optimization is executed (and plans a ScalarSubquery expression).
|
1 2 3 4 5 6 7 8 9 10 |
// FIXME DEMO import org.apache.spark.sql.execution.PlanSubqueries val spark = ... val planSubqueries = PlanSubqueries(spark) val plan = ... val executedPlan = planSubqueries(plan) |
ScalarSubquery expression cannot be evaluated, i.e. produce a value given an internal row.
ScalarSubquery uses…FIXME…for the data type.
| Name | Description |
|---|---|
|
|
The value of the single column in a single row after collecting the rows from executing the subquery plan or |
|
|
Flag that says whether |
Updating ScalarSubquery With Collected Result — updateResult Method
|
1 2 3 4 5 |
updateResult(): Unit |
|
Note
|
updateResult is part of ExecSubqueryExpression Contract to fill an Catalyst expression with a collected result from executing a subquery plan.
|
updateResult requests SubqueryExec physical plan to execute and collect internal rows.
updateResult sets result to the value of the only column of the single row or null if no row were collected.
In the end, updateResult marks the ScalarSubquery instance as updated.
updateResult reports a RuntimeException when there are more than 1 rows in the result.
|
1 2 3 4 5 6 |
more than one row returned by a subquery used as an expression: [plan] |
updateResult reports an AssertionError when the number of fields is not exactly 1.
|
1 2 3 4 5 |
Expects 1 field, but got [numFields] something went wrong in analysis |
Evaluating Expression — eval Method
|
1 2 3 4 5 |
eval(input: InternalRow): Any |
|
Note
|
eval is part of Expression Contract for the interpreted (non-code-generated) expression evaluation, i.e. evaluating a Catalyst expression to a JVM object for a given internal binary row.
|
eval simply returns result value.
eval reports an IllegalArgumentException if the ScalarSubquery expression has not been updated yet.
Generating Java Source Code (ExprCode) For Code-Generated Expression Evaluation — doGenCode Method
|
1 2 3 4 5 |
doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode |
|
Note
|
doGenCode is part of Expression Contract to generate a Java source code (ExprCode) for code-generated expression evaluation.
|
doGenCode first makes sure that the updated flag is on (true). If not, doGenCode throws an IllegalArgumentException exception with the following message:
|
1 2 3 4 5 |
requirement failed: [this] has not finished |
doGenCode then creates a Literal (for the result and the dataType) and simply requests it to generate a Java source code.
