spark技术分享HintInfo
HintInfo takes a single broadcast flag when created.
HintInfo is created when:
-
Dataset.broadcast function is used
-
ResolveBroadcastHintslogical resolution rule is executed (and resolves UnresolvedHint logical operators) -
ResolvedHint and Statistics are created
-
InMemoryRelationis requested for computeStats (when sizeInBytesStats is0) -
HintInfois requested to resetForJoin
broadcast is used to…FIXME
broadcast is off (i.e. false) by default.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
import org.apache.spark.sql.catalyst.plans.logical.HintInfo val broadcastOff = HintInfo() scala> println(broadcastOff.broadcast) false val broadcastOn = broadcastOff.copy(broadcast = true) scala> println(broadcastOn) (broadcast) val broadcastOff = broadcastOn.resetForJoin scala> println(broadcastOff.broadcast) false |
resetForJoin Method
|
1 2 3 4 5 |
resetForJoin(): HintInfo |
resetForJoin…FIXME
|
Note
|
resetForJoin is used when SizeInBytesOnlyStatsPlanVisitor is requested to visitIntersect and visitJoin.
|
