spark技术分享InputProcessor Helper Class of FlatMapGroupsWithStateExec Physical Operator
InputProcessor is a helper class to manage the state store of FlatMapGroupsWithStateExec physical operator.
InputProcessor is created exclusively when FlatMapGroupsWithStateExec physical operator is executed (and uses InputProcessor for the storeUpdateFunction while mapping over partitions with a state store).
InputProcessor takes a single StateStore when created.
processNewData Method
|
1 2 3 4 5 |
processNewData(dataIter: Iterator[InternalRow]): Iterator[InternalRow] |
processNewData…FIXME
|
Note
|
processNewData is used exclusively when FlatMapGroupsWithStateExec physical operator is executed.
|
processTimedOutState Method
|
1 2 3 4 5 |
processTimedOutState(): Iterator[InternalRow] |
processTimedOutState…FIXME
|
Note
|
processTimedOutState is used exclusively when FlatMapGroupsWithStateExec physical operator is executed.
|
callFunctionAndUpdateState Internal Method
|
1 2 3 4 5 6 7 8 |
callFunctionAndUpdateState( stateData: StateData, valueRowIter: Iterator[InternalRow], hasTimedOut: Boolean): Iterator[InternalRow] |
callFunctionAndUpdateState…FIXME
|
Note
|
callFunctionAndUpdateState is used exclusively when InputProcessor is requested to processNewData and processTimedOutState.
|
