OneSideHashJoiner-spark技术分享

OneSideHashJoiner

OneSideHashJoiner is created exclusively when StreamingSymmetricHashJoinExec physical operator is requested to processPartitions (when requested to doExecute).

Note	`OneSideHashJoiner` is a Scala private internal class of StreamingSymmetricHashJoinExec.

Table 1. OneSideHashJoiner’s Internal Properties (e.g. Registries, Counters and Flags)
Name	Description
`joinStateManager`	SymmetricHashJoinStateManager Used when…FIXME
`keyGenerator`	`UnsafeProjection` to generate join keys Used when…FIXME
`preJoinFilter`	`InternalRow ⇒ Boolean` Used when…FIXME
`stateKeyWatermarkPredicateFunc`	`InternalRow ⇒ Boolean` Used when…FIXME
`stateValueWatermarkPredicateFunc`	`InternalRow ⇒ Boolean` Used when…FIXME
`updatedStateRowsCount`	Counter Used exclusively when requested to storeAndJoinWithOtherSide

`storeAndJoinWithOtherSide` Method



storeAndJoinWithOtherSide(
  otherSideJoiner: OneSideHashJoiner)(
  generateJoinedRow: (InternalRow, InternalRow) => JoinedRow): Iterator[InternalRow]

storeAndJoinWithOtherSide(

otherSideJoiner: OneSideHashJoiner)(

generateJoinedRow: (InternalRow, InternalRow) => JoinedRow): Iterator[InternalRow]

storeAndJoinWithOtherSide…FIXME

Note	`storeAndJoinWithOtherSide` is used when `StreamingSymmetricHashJoinExec` physical operator is requested to processPartitions (when `StreamingSymmetricHashJoinExec` physical operator is requested to doExecute).

Creating OneSideHashJoiner Instance

OneSideHashJoiner takes the following to be created:

JoinSide
Input attributes (Seq[Attribute])
Join keys (Seq[Expression])
Input internal rows (Iterator[InternalRow])
Optional pre-join filter Catalyst expression
Post-join filter ((InternalRow) ⇒ Boolean)
Optional JoinStateWatermarkPredicate

OneSideHashJoiner initializes the internal registries and counters.

`removeOldState` Method



removeOldState(): Iterator[UnsafeRowPair]

removeOldState(): Iterator[UnsafeRowPair]

removeOldState…FIXME

Note	`removeOldState` is used when…FIXME

`get` Method



get(key: UnsafeRow): Iterator[UnsafeRow]

get(key: UnsafeRow): Iterator[UnsafeRow]

get…FIXME

Note	`get` is used when…FIXME

`commitStateAndGetMetrics` Method



commitStateAndGetMetrics(): StateStoreMetrics

commitStateAndGetMetrics(): StateStoreMetrics

commitStateAndGetMetrics…FIXME

Note	`commitStateAndGetMetrics` is used when…FIXME

OneSideHashJoiner