spark技术分享ComputeCurrentTime Logical Optimization
ComputeCurrentTime is a base logical optimization that computes the current date and timestamp.
ComputeCurrentTime is part of the Finish Analysis once-executed batch in the standard batches of the Catalyst Optimizer.
ComputeCurrentTime is simply a Catalyst rule for transforming logical plans, i.e. Rule[LogicalPlan].
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
// Query with two current_date's import org.apache.spark.sql.functions.current_date val q = spark.range(1).select(current_date() as "d1", current_date() as "d2") val analyzedPlan = q.queryExecution.analyzed scala> println(analyzedPlan.numberedTreeString) 00 Project [current_date(Some(Europe/Warsaw)) AS d1#12, current_date(Some(Europe/Warsaw)) AS d2#13] 01 +- Range (0, 1, step=1, splits=Some(8)) import org.apache.spark.sql.catalyst.optimizer.ComputeCurrentTime val afterComputeCurrentTime = ComputeCurrentTime(analyzedPlan) scala> println(afterComputeCurrentTime.numberedTreeString) 00 Project [17773 AS d1#12, 17773 AS d2#13] 01 +- Range (0, 1, step=1, splits=Some(8)) // Another query with two current_timestamp's // Here the millis play a bigger role so it is easier to notice the results import org.apache.spark.sql.functions.current_timestamp val q = spark.range(1).select(current_timestamp() as "ts1", current_timestamp() as "ts2") val analyzedPlan = q.queryExecution.analyzed val afterComputeCurrentTime = ComputeCurrentTime(analyzedPlan) scala> println(afterComputeCurrentTime.numberedTreeString) 00 Project [1535629687768000 AS ts1#18, 1535629687768000 AS ts2#19] 01 +- Range (0, 1, step=1, splits=Some(8)) |
Executing Rule — apply Method
|
1 2 3 4 5 |
apply(plan: LogicalPlan): LogicalPlan |
|
Note
|
apply is part of the Rule Contract to execute (apply) a rule on a TreeNode (e.g. LogicalPlan).
|
apply…FIXME
