SubExprUtils Helper Object-spark技术分享

SubExprUtils Helper Object

SubExprUtils is a Scala object that is used for…FIXME

SubExprUtils uses PredicateHelper for…FIXME

SubExprUtils is used to check whether a condition expression has any null-aware predicate subqueries inside Not expressions.

Checking If Condition Expression Has Any Null-Aware Predicate Subqueries Inside Not — `hasNullAwarePredicateWithinNot` Method



hasNullAwarePredicateWithinNot(condition: Expression): Boolean

hasNullAwarePredicateWithinNot(condition: Expression): Boolean

hasNullAwarePredicateWithinNot splits conjunctive predicates (i.e. expressions separated by And expression).

hasNullAwarePredicateWithinNot is positive (i.e. true) and is considered to have a null-aware predicate subquery inside a Not expression when conjuctive predicate expressions include a Not expression with an In predicate expression with a ListQuery subquery expression.



import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.dsl.plans._
val plan = LocalRelation('key.int, 'value.string).analyze

import org.apache.spark.sql.catalyst.expressions._
val in = In(value = Literal.create(1), Seq(ListQuery(plan)))
val condition = Not(child = Or(left = Literal.create(false), right = in))

import org.apache.spark.sql.catalyst.expressions.SubExprUtils
val positive = SubExprUtils.hasNullAwarePredicateWithinNot(condition)
assert(positive)

import org.apache.spark.sql.catalyst.plans.logical._

import org.apache.spark.sql.catalyst.dsl.plans._

val plan = LocalRelation('key.int, 'value.string).analyze

import org.apache.spark.sql.catalyst.expressions._

val in = In(value = Literal.create(1), Seq(ListQuery(plan)))

val condition = Not(child = Or(left = Literal.create(false), right = in))

import org.apache.spark.sql.catalyst.expressions.SubExprUtils

val positive = SubExprUtils.hasNullAwarePredicateWithinNot(condition)

assert(positive)

hasNullAwarePredicateWithinNot is negative (i.e. false) for all the other expressions and in particular the following expressions:

Exists predicate subquery expressions
Not expressions with a Exists predicate subquery expression as the child expression
In expressions with a ListQuery subquery expression as the list expression
Not expressions with a In expression (with a ListQuery subquery expression as the list expression)



import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.dsl.plans._
val plan = LocalRelation('key.int, 'value.string).analyze

import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.expressions.SubExprUtils

// Exists
val condition = Exists(plan)
val negative = SubExprUtils.hasNullAwarePredicateWithinNot(condition)
assert(!negative)

// Not Exists
val condition = Not(child = Exists(plan))
val negative = SubExprUtils.hasNullAwarePredicateWithinNot(condition)
assert(!negative)

// In with ListQuery
val condition = In(value = Literal.create(1), Seq(ListQuery(plan)))
val negative = SubExprUtils.hasNullAwarePredicateWithinNot(condition)
assert(!negative)

// Not In with ListQuery
val in = In(value = Literal.create(1), Seq(ListQuery(plan)))
val condition = Not(child = in)
val negative = SubExprUtils.hasNullAwarePredicateWithinNot(condition)
assert(!negative)

import org.apache.spark.sql.catalyst.plans.logical._

import org.apache.spark.sql.catalyst.dsl.plans._

val plan = LocalRelation('key.int, 'value.string).analyze

import org.apache.spark.sql.catalyst.expressions._

import org.apache.spark.sql.catalyst.expressions.SubExprUtils

// Exists

val condition = Exists(plan)

val negative = SubExprUtils.hasNullAwarePredicateWithinNot(condition)

assert(!negative)

// Not Exists

val condition = Not(child = Exists(plan))

val negative = SubExprUtils.hasNullAwarePredicateWithinNot(condition)

assert(!negative)

// In with ListQuery

val condition = In(value = Literal.create(1), Seq(ListQuery(plan)))

val negative = SubExprUtils.hasNullAwarePredicateWithinNot(condition)

assert(!negative)

// Not In with ListQuery

val in = In(value = Literal.create(1), Seq(ListQuery(plan)))

val condition = Not(child = in)

val negative = SubExprUtils.hasNullAwarePredicateWithinNot(condition)

assert(!negative)

Note	`hasNullAwarePredicateWithinNot` is used exclusively when `CheckAnalysis` analysis validation is requested to validate analysis of a logical plan (with `Filter` logical operators).

SubExprUtils Helper Object