关注 spark技术分享,
撸spark源码 玩spark最佳实践

ExtractPythonUDFs

ExtractPythonUDFs Physical Query Optimization

ExtractPythonUDFs is a physical query optimization (aka physical query preparation rule or simply preparation rule) that QueryExecution uses to optimize the physical plan of a structured query by extracting Python UDFs from a physical query plan (excluding FlatMapGroupsInPandasExec operators that it simply skips over).

Technically, ExtractPythonUDFs is just a Catalyst rule for transforming physical query plans, i.e. Rule[SparkPlan].

ExtractPythonUDFs is part of preparations batch of physical query plan rules and is executed when QueryExecution is requested for the optimized physical query plan (i.e. in executedPlan phase of a query execution).

Extracting Python UDFs from Physical Query Plan — extract Internal Method

extract…​FIXME

Note
extract is used exclusively when ExtractPythonUDFs is requested to optimize a physical query plan.

trySplitFilter Internal Method

trySplitFilter…​FIXME

Note
trySplitFilter is used exclusively when ExtractPythonUDFs is requested to extract.
赞(0) 打赏
未经允许不得转载:spark技术分享 » ExtractPythonUDFs
分享到: 更多 (0)

关注公众号:spark技术分享

联系我们联系我们

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏