spark技术分享ParseToTimestamp Expression
ParseToTimestamp is a RuntimeReplaceable expression that represents the to_timestamp function (in logical query plans).
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
// DEMO to_timestamp(s: Column): Column import java.sql.Timestamp import java.time.LocalDateTime val times = Seq(Timestamp.valueOf(LocalDateTime.of(2018, 5, 30, 0, 0, 0)).toString).toDF("time") scala> times.printSchema root |-- time: string (nullable = true) import org.apache.spark.sql.functions.to_timestamp val q = times.select(to_timestamp($"time") as "ts") scala> q.printSchema root |-- ts: timestamp (nullable = true) val plan = q.queryExecution.analyzed scala> println(plan.numberedTreeString) 00 Project [to_timestamp('time, None) AS ts#29] 01 +- Project [value#16 AS time#18] 02 +- LocalRelation [value#16] import org.apache.spark.sql.catalyst.expressions.ParseToTimestamp val ptt = plan.expressions.head.children.head.asInstanceOf[ParseToTimestamp] scala> println(ptt.numberedTreeString) 00 to_timestamp('time, None) 01 +- cast(time#18 as timestamp) 02 +- time#18: string // FIXME DEMO to_timestamp(s: Column, fmt: String): Column |
As a RuntimeReplaceable expression, ParseToTimestamp is replaced by Catalyst Optimizer with the child expression:
-
Cast(left, TimestampType)forto_timestamp(s: Column): Columnfunction -
Cast(UnixTimestamp(left, format), TimestampType)forto_timestamp(s: Column, fmt: String): Columnfunction
|
1 2 3 4 5 6 |
// FIXME DEMO Conversion to `Cast(left, TimestampType)` // FIXME DEMO Conversion to `Cast(UnixTimestamp(left, format), TimestampType)` |
Creating ParseToTimestamp Instance
ParseToTimestamp takes the following when created:
-
Left expression
-
formatexpression -
Child expression
