UnsafeProjection — Generic Function to Project InternalRows to UnsafeRows-spark技术分享

UnsafeProjection — Generic Function to Encode InternalRows to UnsafeRows

UnsafeProjection is a Projection function that encodes InternalRows as UnsafeRows.



UnsafeProjection: InternalRow =[apply]=> UnsafeRow

UnsafeProjection: InternalRow =[apply]=> UnsafeRow

Note	Spark SQL uses `UnsafeProjection` factory object to create concrete adhoc `UnsafeProjection` instances. The base `UnsafeProjection` has no concrete named implementations and create factory methods delegate all calls to GenerateUnsafeProjection.generate in the end.

Creating UnsafeProjection — `create` Factory Method



create(schema: StructType): UnsafeProjection      (1)
create(fields: Array[DataType]): UnsafeProjection (2)
create(expr: Expression): UnsafeProjection        (3)
create(exprs: Seq[Expression], inputSchema: Seq[Attribute]): UnsafeProjection (4)
create(exprs: Seq[Expression]): UnsafeProjection  (5)
create(
  exprs: Seq[Expression],
  inputSchema: Seq[Attribute],
  subexpressionEliminationEnabled: Boolean): UnsafeProjection

create(schema: StructType): UnsafeProjection (1)

create(fields: Array[DataType]): UnsafeProjection (2)

create(expr: Expression): UnsafeProjection (3)

create(exprs: Seq[Expression], inputSchema: Seq[Attribute]): UnsafeProjection (4)

create(exprs: Seq[Expression]): UnsafeProjection (5)

create(

exprs: Seq[Expression],

inputSchema: Seq[Attribute],

subexpressionEliminationEnabled: Boolean): UnsafeProjection

create takes the DataTypes from schema and calls the 2nd create
create creates a BoundReference per field in fields and calls the 5th create
create calls the 5th create
create calls the 5th create
The main create that does the heavy work

create transforms all CreateNamedStruct expressions to CreateNamedStructUnsafe in every BoundReference in the input exprs.

In the end, create requests GenerateUnsafeProjection to generate a UnsafeProjection.

Note	A variant of `create` takes `subexpressionEliminationEnabled` flag (that usually is subexpressionEliminationEnabled flag of `SparkPlan`).

UnsafeProjection — Generic Function to Project InternalRows to UnsafeRows

UnsafeProjection — Generic Function to Encode InternalRows to UnsafeRows

Creating UnsafeProjection — `create` Factory Method

相关推荐

欢迎关注：spark技术分享

热门标签

近期文章

分类目录

关注公众号：spark技术分享

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏

QQ咨询

回顶部

UnsafeProjection — Generic Function to Encode InternalRows to UnsafeRows

Creating UnsafeProjection — create Factory Method

相关推荐

欢迎关注：spark技术分享

热门标签

近期文章

分类目录

关注公众号：spark技术分享

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏

QQ咨询

回顶部

Creating UnsafeProjection — `create` Factory Method