spark技术分享ScalaReflection
ScalaReflection is the contract and the only implementation of the contract with…FIXME
serializerFor Object Method
|
1 2 3 4 5 |
serializerFor[T : TypeTag](inputObject: Expression): CreateNamedStruct |
serializerFor firstly finds the local type of the input type T and then the class name.
serializerFor uses the internal version of itself with the input inputObject expression, the tpe type and the walkedTypePath with the class name found earlier (of the input type T).
|
1 2 3 4 5 |
- root class: "[clsName]" |
In the end, serializerFor returns one of the following:
-
The CreateNamedStruct expression from the false value of the
Ifexpression returned only if the typeTis definedByConstructorParams -
Creates a CreateNamedStruct expression with the Literal with the value as
"value"and the expression returned
|
1 2 3 4 5 6 7 8 9 10 11 |
import org.apache.spark.sql.functions.lit val inputObject = lit(1).expr import org.apache.spark.sql.catalyst.ScalaReflection val serializer = ScalaReflection.serializerFor(inputObject) scala> println(serializer) named_struct(value, 1) |
|
Note
|
serializerFor is used when…FIXME
|
serializerFor Internal Method
|
1 2 3 4 5 6 7 8 9 |
serializerFor( inputObject: Expression, tpe: `Type`, walkedTypePath: Seq[String], seenTypeSet: Set[`Type`] = Set.empty): Expression |
serializerFor…FIXME
|
Note
|
serializerFor is used exclusively when ScalaReflection is requested to serializerFor.
|
localTypeOf Object Method
|
1 2 3 4 5 |
localTypeOf[T: TypeTag]: `Type` |
localTypeOf…FIXME
|
1 2 3 4 5 6 7 8 9 10 11 |
import org.apache.spark.sql.catalyst.ScalaReflection val tpe = ScalaReflection.localTypeOf[Int] scala> :type tpe org.apache.spark.sql.catalyst.ScalaReflection.universe.Type scala> println(tpe) Int |
|
Note
|
localTypeOf is used when…FIXME
|
getClassNameFromType Object Method
|
1 2 3 4 5 |
getClassNameFromType(tpe: `Type`): String |
getClassNameFromType…FIXME
|
1 2 3 4 5 6 7 8 9 |
import org.apache.spark.sql.catalyst.ScalaReflection val tpe = ScalaReflection.localTypeOf[java.time.LocalDateTime] val className = ScalaReflection.getClassNameFromType(tpe) scala> println(className) java.time.LocalDateTime |
|
Note
|
getClassNameFromType is used when…FIXME
|
