spark技术分享Nondeterministic Expression Contract
Nondeterministic is a contract for Catalyst expressions that are non-deterministic and non-foldable.
Nondeterministic expressions require explicit initialization (with the current partition index) before evaluating a value.
|
1 2 3 4 5 6 7 8 9 10 11 |
package org.apache.spark.sql.catalyst.expressions trait Nondeterministic extends Expression { // only required methods that have no implementation protected def initializeInternal(partitionIndex: Int): Unit protected def evalInternal(input: InternalRow): Any } |
| Method | Description |
|---|---|
|
|
Initializing the Used exclusively when |
|
|
Evaluating the Used exclusively when |
|
Note
|
Nondeterministic expressions are the target of PullOutNondeterministic logical plan rule.
|
| Expression | Description |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| Name | Description |
|---|---|
|
Always turned off (i.e. |
|
|
Always turned off (i.e. |
|
|
Controls whether a Turned off by default. |
Initializing Expression — initialize Method
|
1 2 3 4 5 |
initialize(partitionIndex: Int): Unit |
Internally, initialize initializes itself (with the input partition index) and turns the internal initialized flag on.
|
Note
|
initialize is used exclusively when InterpretedProjection and InterpretedMutableProjection are requested to initialize themselves.
|
Evaluating Expression — eval Method
|
1 2 3 4 5 |
eval(input: InternalRow): Any |
|
Note
|
eval is part of Expression Contract for the interpreted (non-code-generated) expression evaluation, i.e. evaluating a Catalyst expression to a JVM object for a given internal binary row.
|
eval is just a wrapper of evalInternal that makes sure that initialize has already been executed (and so the expression is initialized).
Internally, eval makes sure that the expression was initialized and calls evalInternal.
eval reports a IllegalArgumentException exception when the internal initialized flag is off, i.e. initialize has not yet been executed.
|
1 2 3 4 5 |
requirement failed: Nondeterministic expression [name] should be initialized before eval. |
