Catalog — Metastore Management Interface-spark技术分享

Catalog — Metastore Management Interface

Catalog is the interface for managing a metastore (aka metadata catalog) of relational entities (e.g. database(s), tables, functions, table columns and temporary views).

Catalog is available using SparkSession.catalog property.



scala> :type spark
org.apache.spark.sql.SparkSession

scala> :type spark.catalog
org.apache.spark.sql.catalog.Catalog

scala> :type spark

org.apache.spark.sql.SparkSession

scala> :type spark.catalog

org.apache.spark.sql.catalog.Catalog

Method Description

cacheTable



cacheTable(tableName: String): Unit
cacheTable(tableName: String, storageLevel: StorageLevel): Unit

cacheTable(tableName: String): Unit

cacheTable(tableName: String, storageLevel: StorageLevel): Unit

Caches the specified table in memory

Used for SQL’s CACHE TABLE and AlterTableRenameCommand command.

clearCache



clearCache(): Unit

clearCache(): Unit

createTable



createTable(tableName: String, path: String): DataFrame
createTable(
  tableName: String,
  source: String,
  options: java.util.Map[String, String]): DataFrame
createTable(
  tableName: String,
  source: String,
  options: Map[String, String]): DataFrame
createTable(tableName: String, path: String, source: String): DataFrame
createTable(
  tableName: String,
  source: String,
  schema: StructType,
  options: java.util.Map[String, String]): DataFrame
createTable(
  tableName: String,
  source: String,
  schema: StructType,
  options: Map[String, String]): DataFrame

createTable(tableName: String, path: String): DataFrame

createTable(

tableName: String,

source: String,

options: java.util.Map[String, String]): DataFrame

createTable(

tableName: String,

source: String,

options: Map[String, String]): DataFrame

createTable(tableName: String, path: String, source: String): DataFrame

createTable(

tableName: String,

source: String,

schema: StructType,

options: java.util.Map[String, String]): DataFrame

createTable(

tableName: String,

source: String,

schema: StructType,

options: Map[String, String]): DataFrame

currentDatabase



currentDatabase: String

currentDatabase: String

databaseExists



databaseExists(dbName: String): Boolean

databaseExists(dbName: String): Boolean

dropGlobalTempView



dropGlobalTempView(viewName: String): Boolean

dropGlobalTempView(viewName: String): Boolean

dropTempView



dropTempView(viewName: String): Boolean

dropTempView(viewName: String): Boolean

functionExists



functionExists(functionName: String): Boolean
functionExists(dbName: String, functionName: String): Boolean

functionExists(functionName: String): Boolean

functionExists(dbName: String, functionName: String): Boolean

getDatabase



getDatabase(dbName: String): Database

getDatabase(dbName: String): Database

getFunction



getFunction(functionName: String): Function
getFunction(dbName: String, functionName: String): Function

getFunction(functionName: String): Function

getFunction(dbName: String, functionName: String): Function

getTable



getTable(tableName: String): Table
getTable(dbName: String, tableName: String): Table

getTable(tableName: String): Table

getTable(dbName: String, tableName: String): Table

isCached



isCached(tableName: String): Boolean

isCached(tableName: String): Boolean

listColumns



listColumns(tableName: String): Dataset[Column]
listColumns(dbName: String, tableName: String): Dataset[Column]

listColumns(tableName: String): Dataset[Column]

listColumns(dbName: String, tableName: String): Dataset[Column]

listDatabases



listDatabases(): Dataset[Database]

listDatabases(): Dataset[Database]

listFunctions



listFunctions(): Dataset[Function]
listFunctions(dbName: String): Dataset[Function]

listFunctions(): Dataset[Function]

listFunctions(dbName: String): Dataset[Function]

listTables



listTables(): Dataset[Table]
listTables(dbName: String): Dataset[Table]

listTables(): Dataset[Table]

listTables(dbName: String): Dataset[Table]

recoverPartitions



recoverPartitions(tableName: String): Unit

recoverPartitions(tableName: String): Unit

refreshByPath



refreshByPath(path: String): Unit

refreshByPath(path: String): Unit

refreshTable



refreshTable(tableName: String): Unit

refreshTable(tableName: String): Unit

setCurrentDatabase



setCurrentDatabase(dbName: String): Unit

setCurrentDatabase(dbName: String): Unit

tableExists



tableExists(tableName: String): Boolean
tableExists(dbName: String, tableName: String): Boolean

tableExists(tableName: String): Boolean

tableExists(dbName: String, tableName: String): Boolean

uncacheTable



uncacheTable(tableName: String): Unit

uncacheTable(tableName: String): Unit

Note	CatalogImpl is the one and only known implementation of the Catalog Contract in Apache Spark.

Catalog — Metastore Management Interface

Catalog — Metastore Management Interface

相关推荐

欢迎关注：spark技术分享

热门标签

近期文章

分类目录

关注公众号：spark技术分享

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏

QQ咨询

回顶部