FunctionRegistry — Contract for Function Registries (Catalogs)-spark技术分享

FunctionRegistry — Contract for Function Registries (Catalogs)

FunctionRegistry is the contract of function registries (catalogs) of native and user-defined functions.



package org.apache.spark.sql.catalyst.analysis

trait FunctionRegistry {
  // only required properties (vals and methods) that have no implementation
  // the others follow
  def clear(): Unit
  def dropFunction(name: FunctionIdentifier): Boolean
  def listFunction(): Seq[FunctionIdentifier]
  def lookupFunction(name: FunctionIdentifier): Option[ExpressionInfo]
  def lookupFunction(name: FunctionIdentifier, children: Seq[Expression]): Expression
  def lookupFunctionBuilder(name: FunctionIdentifier): Option[FunctionBuilder]
  def registerFunction(
    name: FunctionIdentifier,
    info: ExpressionInfo,
    builder: FunctionBuilder): Unit
}

package org.apache.spark.sql.catalyst.analysis

trait FunctionRegistry {

// only required properties (vals and methods) that have no implementation

// the others follow

def clear(): Unit

def dropFunction(name: FunctionIdentifier): Boolean

def listFunction(): Seq[FunctionIdentifier]

def lookupFunction(name: FunctionIdentifier): Option[ExpressionInfo]

def lookupFunction(name: FunctionIdentifier, children: Seq[Expression]): Expression

def lookupFunctionBuilder(name: FunctionIdentifier): Option[FunctionBuilder]

def registerFunction(

name: FunctionIdentifier,

info: ExpressionInfo,

builder: FunctionBuilder): Unit

}

Table 1. FunctionRegistry Contract
Property	Description
`clear`	Used exclusively when `SessionCatalog` is requested to reset
`dropFunction`	Used when…FIXME
`listFunction`	Used when…FIXME
`lookupFunction`	Used when: `FunctionRegistry` is requested to functionExists `SessionCatalog` is requested to find a function by name, lookupFunctionInfo or reset `HiveSessionCatalog` is requested to lookupFunction0
`lookupFunctionBuilder`	Used when…FIXME
`registerFunction`	Used when: `SessionCatalog` is requested to registerFunction or reset `FunctionRegistry` is requested for a SimpleFunctionRegistry with the built-in functions registered or createOrReplaceTempFunction `SimpleFunctionRegistry` is requested to `clone`

Note	The one and only `FunctionRegistry` available in Spark SQL is SimpleFunctionRegistry.

FunctionRegistry is available through functionRegistry property of a SessionState (that is available as sessionState property of a SparkSession).



scala> :type spark
org.apache.spark.sql.SparkSession

scala> :type spark.sessionState.functionRegistry
org.apache.spark.sql.catalyst.analysis.FunctionRegistry

scala> :type spark

org.apache.spark.sql.SparkSession

scala> :type spark.sessionState.functionRegistry

org.apache.spark.sql.catalyst.analysis.FunctionRegistry

Note	You can register a new user-defined function using UDFRegistration.

Table 2. FunctionRegistry’s Attributes
Name	Description
`builtin`	SimpleFunctionRegistry with the built-in functions registered.

FunctionRegistry manages function expression registry of Catalyst expressions and the corresponding built-in/native SQL functions (that can be used in SQL statements).

Table 3. (Subset of) FunctionRegistry’s Catalyst Expression to SQL Function Mapping
Catalyst Expression	SQL Function
CumeDist	`cume_dist`
`IfNull`	`ifnull`
`Left`	`left`
MonotonicallyIncreasingID	`monotonically_increasing_id`
`NullIf`	`nullif`
`Nvl`	`nvl`
`Nvl2`	`nvl2`
ParseToDate	`to_date`
ParseToTimestamp	`to_timestamp`
`Right`	`right`
CreateNamedStruct	`struct`

`expression` Internal Method



expression[T <: Expression](name: String)
  (implicit tag: ClassTag[T]): (String, (ExpressionInfo, FunctionBuilder))

expression[T <: Expression](name: String)

(implicit tag: ClassTag[T]): (String, (ExpressionInfo, FunctionBuilder))

expression…FIXME

Note	`expression` is used when…FIXME

SimpleFunctionRegistry

SimpleFunctionRegistry is the default FunctionRegistry that is backed by a hash map (with optional case sensitivity).

`createOrReplaceTempFunction` Final Method



createOrReplaceTempFunction(name: String, builder: FunctionBuilder): Unit

createOrReplaceTempFunction(name: String, builder: FunctionBuilder): Unit

createOrReplaceTempFunction…FIXME

Note	`createOrReplaceTempFunction` is used exclusively when `UDFRegistration` is requested to register an user-defined function, user-defined aggregate function, user-defined function (as UserDefinedFunction) or `registerPython`.

`functionExists` Method



functionExists(name: FunctionIdentifier): Boolean

functionExists(name: FunctionIdentifier): Boolean

functionExists…FIXME

Note	`functionExists` is used when…FIXME

FunctionRegistry — Contract for Function Registries (Catalogs)