CheckAnalysis — Analysis Validation-spark技术分享

Validating Analysis of Logical Plan (and Marking Plan As Analyzed) — `checkAnalysis` Method



checkAnalysis(plan: LogicalPlan): Unit

1

2

3

4

5

checkAnalysis(plan: LogicalPlan): Unit

checkAnalysis recursively checks the correctness of the analysis of the input logical plan and marks it as analyzed.

Note	`checkAnalysis` fails analysis when finds UnresolvedRelation in the input `LogicalPlan`…FIXME What else?

Internally, checkAnalysis processes nodes in the input plan (starting from the leafs, i.e. nodes down the operator tree).

checkAnalysis skips logical plans that have already undergo analysis.

LogicalPlan/Operator Behaviour

UnresolvedRelation

Fails analysis with the error message:



Table or view not found: [tableIdentifier]

1

2

3

4

5

Table or view not found: [tableIdentifier]

Unresolved Attribute

Fails analysis with the error message:



cannot resolve '[expr]' given input columns: [from]

1

2

3

4

5

cannot resolve '[expr]' given input columns: [from]

Expression with incorrect input data types

Fails analysis with the error message:



cannot resolve '[expr]' due to data type mismatch: [message]

1

2

3

4

5

cannot resolve '[expr]' due to data type mismatch: [message]

Unresolved Cast

Fails analysis with the error message:



invalid cast from [dataType] to [dataType]

1

2

3

4

5

invalid cast from [dataType] to [dataType]

Grouping

Fails analysis with the error message:



grouping() can only be used with GroupingSets/Cube/Rollup

1

2

3

4

5

grouping() can only be used with GroupingSets/Cube/Rollup

GroupingID

Fails analysis with the error message:



grouping_id() can only be used with GroupingSets/Cube/Rollup

1

2

3

4

5

grouping_id() can only be used with GroupingSets/Cube/Rollup

WindowExpressions with a AggregateExpression window function with isDistinct flag on

Fails analysis with the error message:



Distinct window functions are not supported: [w]

1

2

3

4

5

Distinct window functions are not supported: [w]

Example:



val windowedDistinctCountExpr = "COUNT(DISTINCT 1) OVER (PARTITION BY value)"
scala> spark.emptyDataset[Int].selectExpr(windowedDistinctCountExpr)
org.apache.spark.sql.AnalysisException: Distinct window functions are not supported: count(distinct 1) windowspecdefinition(value#95, ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING);;
Project [COUNT(1) OVER (PARTITION BY value UnspecifiedFrame)#97L]
+- Project [value#95, COUNT(1) OVER (PARTITION BY value UnspecifiedFrame)#97L, COUNT(1) OVER (PARTITION BY value UnspecifiedFrame)#97L]
   +- Window [count(distinct 1) windowspecdefinition(value#95, ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS COUNT(1) OVER (PARTITION BY value UnspecifiedFrame)#97L], [value#95]
      +- Project [value#95]
         +- LocalRelation <empty>, [value#95]

  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:40)
  at org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:90)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:108)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:86)

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

val windowedDistinctCountExpr = "COUNT(DISTINCT 1) OVER (PARTITION BY value)"

scala> spark.emptyDataset[Int].selectExpr(windowedDistinctCountExpr)

org.apache.spark.sql.AnalysisException: Distinct window functions are not supported: count(distinct 1) windowspecdefinition(value#95, ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING);;

Project [COUNT(1) OVER (PARTITION BY value UnspecifiedFrame)#97L]

+- Project [value#95, COUNT(1) OVER (PARTITION BY value UnspecifiedFrame)#97L, COUNT(1) OVER (PARTITION BY value UnspecifiedFrame)#97L]

+- Window [count(distinct 1) windowspecdefinition(value#95, ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS COUNT(1) OVER (PARTITION BY value UnspecifiedFrame)#97L], [value#95]

+- Project [value#95]

+- LocalRelation <empty>, [value#95]

at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:40)

at org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:90)

at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:108)

at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:86)

WindowExpressions with a OffsetWindowFunction window function with an empty order specification or a non-offset window frame specification

Fails analysis with the error message:



An offset window function can only be evaluated in an ordered row-based window frame with a single offset: [windowExpr]

1

2

3

4

5

An offset window function can only be evaluated in an ordered row-based window frame with a single offset: [windowExpr]

WindowExpressions with a window function that is not one of the following expressions: AggregateExpression, AggregateWindowFunction or OffsetWindowFunction

Fails analysis with the error message:



Expression '[e]' not supported within a window function.

1

2

3

4

5

Expression '[e]' not supported within a window function.

Nondeterministic expressions

FIXME

UnresolvedHint

FIXME

After the validations, checkAnalysis executes additional check rules for correct analysis.

checkAnalysis then checks if plan is analyzed correctly (i.e. no logical plans are left unresolved). If there is one, checkAnalysis fails the analysis with AnalysisException and the following error message:



unresolved operator [o.simpleString]

1

2

3

4

5

unresolved operator [o.simpleString]

In the end, checkAnalysis marks the entire logical plan as analyzed.

Note	`checkAnalysis` is used when: `QueryExecution` creates analyzed logical plan and checks its correctness (which happens mostly when a `Dataset` is created) `ExpressionEncoder` does resolveAndBind `ResolveAggregateFunctions` is executed (for `Sort` logical plan)

Extended Analysis Check Rules — `extendedCheckRules` Extension Point



extendedCheckRules: Seq[LogicalPlan => Unit]

1

2

3

4

5

extendedCheckRules: Seq[LogicalPlan => Unit]

extendedCheckRules is a collection of rules (functions) that checkAnalysis uses for custom analysis checks (after the main validations have been executed).

Note	When a condition of a rule does not hold the function throws an `AnalysisException` directly or using `failAnalysis` method.

`checkSubqueryExpression` Internal Method



checkSubqueryExpression(plan: LogicalPlan, expr: SubqueryExpression): Unit

1

2

3

4

5

checkSubqueryExpression(plan: LogicalPlan, expr: SubqueryExpression): Unit

checkSubqueryExpression…FIXME

Note	`checkSubqueryExpression` is used exclusively when `CheckAnalysis` is requested to validate analysis of a logical plan (for `SubqueryExpression` expressions).

CheckAnalysis — Analysis Validation