LongHashedRelation-spark技术分享

LongHashedRelation

LongHashedRelation is a HashedRelation that is used when HashedRelation is requested for a concrete HashedRelation instance when the single key is of type long.

LongHashedRelation is also a Java Externalizable, i.e. when persisted, only the identity is written in the serialization stream and it is the responsibility of the class to save and restore the contents of its instances.

LongHashedRelation is created when:

HashedRelation is requested for a concrete HashedRelation (and apply factory method is used)
LongHashedRelation is requested for a read-only copy (when BroadcastHashJoinExec is requested to execute)

`writeExternal` Method



writeExternal(out: ObjectOutput): Unit

writeExternal(out: ObjectOutput): Unit

Note	`writeExternal` is part of Java’s Externalizable Contract to…FIXME.

writeExternal…FIXME

Note	`writeExternal` is used when…FIXME

`readExternal` Method



readExternal(in: ObjectInput): Unit

readExternal(in: ObjectInput): Unit

Note	`readExternal` is part of Java’s Externalizable Contract to…FIXME.

readExternal…FIXME

Note	`readExternal` is used when…FIXME

Creating LongHashedRelation Instance

LongHashedRelation takes the following when created:

Number of fields
LongToUnsafeRowMap

LongHashedRelation initializes the internal registries and counters.

Creating Read-Only Copy of LongHashedRelation — `asReadOnlyCopy` Method



asReadOnlyCopy(): LongHashedRelation

asReadOnlyCopy(): LongHashedRelation

Note	`asReadOnlyCopy` is part of HashedRelation Contract to…FIXME.

asReadOnlyCopy…FIXME

Getting Value Row for Given Key — `getValue` Method



getValue(key: InternalRow): InternalRow

getValue(key: InternalRow): InternalRow

Note	`getValue` is part of HashedRelation Contract to give the value internal row for a given key.

getValue checks if the input key is null at 0 position and if so gives null. Otherwise, getValue takes the long value at position 0 and gets the value.

Creating LongHashedRelation Instance — `apply` Factory Method



apply(
  input: Iterator[InternalRow],
  key: Seq[Expression],
  sizeEstimate: Int,
  taskMemoryManager: TaskMemoryManager): LongHashedRelation

apply(

input: Iterator[InternalRow],

key: Seq[Expression],

sizeEstimate: Int,

taskMemoryManager: TaskMemoryManager): LongHashedRelation

apply…FIXME

Note	`apply` is used exclusively when `HashedRelation` is requested for a concrete HashedRelation.

LongHashedRelation