FlatMapGroupsWithState Unary Logical Operator-spark技术分享

FlatMapGroupsWithState Unary Logical Operator

FlatMapGroupsWithState is a unary logical operator (i.e. LogicalPlan) that is created to represent the following operators in KeyValueGroupedDataset:

Note

FlatMapGroupsWithState is translated to:

FlatMapGroupsWithStateExec physical operator in FlatMapGroupsWithStateStrategy execution planning strategy for streaming Datasets (aka streaming plans)
MapGroupsExec physical operator in BasicOperators execution planning strategy for non-streaming/batch Datasets (aka batch plans)

Creating SerializeFromObject with FlatMapGroupsWithState — `apply` Factory Method



apply[K: Encoder, V: Encoder, S: Encoder, U: Encoder](
  func: (Any, Iterator[Any], LogicalGroupState[Any]) => Iterator[Any],
  groupingAttributes: Seq[Attribute],
  dataAttributes: Seq[Attribute],
  outputMode: OutputMode,
  isMapGroupsWithState: Boolean,
  timeout: GroupStateTimeout,
  child: LogicalPlan): LogicalPlan

apply[K: Encoder, V: Encoder, S: Encoder, U: Encoder](

func: (Any, Iterator[Any], LogicalGroupState[Any]) => Iterator[Any],

groupingAttributes: Seq[Attribute],

dataAttributes: Seq[Attribute],

outputMode: OutputMode,

isMapGroupsWithState: Boolean,

timeout: GroupStateTimeout,

child: LogicalPlan): LogicalPlan

apply creates a SerializeFromObject logical operator with a FlatMapGroupsWithState as its child logical operator.

Internally, apply creates SerializeFromObject object consumer (aka unary logical operator) with FlatMapGroupsWithState logical plan.

Internally, apply finds ExpressionEncoder for the type S and creates a FlatMapGroupsWithState with UnresolvedDeserializer for the types K and V.

In the end, apply creates a SerializeFromObject object consumer with the FlatMapGroupsWithState.

Note	`apply` is used when flatMapGroupsWithState is executed.

Creating FlatMapGroupsWithState Instance

FlatMapGroupsWithState takes the following when created:

State function of type (Any, Iterator[Any], LogicalGroupState[Any]) ⇒ Iterator[Any]
Key deserializer Catalyst expression
Value deserializer Catalyst expression
Grouping attributes
Data attributes
Output object attribute
State ExpressionEncoder
Output mode
isMapGroupsWithState Flag (disabled by default)
GroupStateTimeout
Child logical operator

FlatMapGroupsWithState Unary Logical Operator