spark技术分享AnalysisBarrier Leaf Logical Operator — Hiding Child Query Plan in Analysis
AnalysisBarrier is a leaf logical operator that is a wrapper of an analyzed logical plan to hide it from the Spark Analyzer. The purpose of AnalysisBarrier is to prevent the child logical plan from being analyzed again (and increasing the time spent on query analysis).
AnalysisBarrier is created when:
-
ResolveReferenceslogical resolution rule is requested to dedupRight -
ResolveMissingReferenceslogical resolution rule is requested to resolveExprsAndAddMissingAttrs -
Datasetis created -
DataFrameWriteris requested to execute a logical command for writing to a data source V1 (whenDataFrameWriteris requested to save the rows of a structured query (a DataFrame) to a data source) -
KeyValueGroupedDatasetis requested for the logical query plan
AnalysisBarrier takes a single child logical query plan when created.
AnalysisBarrier returns the child logical query plan when requested for the inner nodes (that should be shown as an inner nested tree of this node).
AnalysisBarrier simply requests the child logical query plan for the output schema attributes.
AnalysisBarrier simply requests the child logical query plan for the isStreaming flag.
AnalysisBarrier simply requests the child logical operator for the canonicalized version.
