StructField — Single Field in StructType-spark技术分享

StructField — Single Field in StructType

StructField describes a single field in a StructType with the following:

Name
DataType
nullable flag (enabled by default)
Metadata (empty by default)

A comment is part of metadata under comment key and is used to build a Hive column or when describing a table.



scala> schemaTyped("a").getComment
res0: Option[String] = None

scala> schemaTyped("a").withComment("this is a comment").getComment
res1: Option[String] = Some(this is a comment)

scala> schemaTyped("a").getComment

res0: Option[String] = None

scala> schemaTyped("a").withComment("this is a comment").getComment

res1: Option[String] = Some(this is a comment)

As of Spark 2.4.0, StructField can be converted to DDL format using toDDL method.


Example: Using StructField.toDDL

import org.apache.spark.sql.types.MetadataBuilder
val metadata = new MetadataBuilder()
  .putString("comment", "this is a comment")
  .build
import org.apache.spark.sql.types.{LongType, StructField}
val f = new StructField(name = "id", dataType = LongType, nullable = false, metadata)
scala> println(f.toDDL)
`id` BIGINT COMMENT 'this is a comment'

Example: Using StructField.toDDL

import org.apache.spark.sql.types.MetadataBuilder

val metadata = new MetadataBuilder()

.putString("comment", "this is a comment")

.build

import org.apache.spark.sql.types.{LongType, StructField}

val f = new StructField(name = "id", dataType = LongType, nullable = false, metadata)

scala> println(f.toDDL)

`id` BIGINT COMMENT 'this is a comment'

Converting to DDL Format — `toDDL` Method



toDDL: String

toDDL: String

toDDL gives a text in the format:



[quoted name] [dataType][optional comment]

[quoted name] [dataType][optional comment]

Note	`toDDL` is used when: `StructType` is requested to convert itself to DDL format ShowCreateTableCommand logical command is executed (and showHiveTableHeader, showHiveTableNonDataColumns, showDataSourceTableDataColumns)

StructField — Single Field in StructType