RunnableCommand Contract — Generic Logical Command with Side Effects-spark技术分享

RunnableCommand Contract — Generic Logical Command with Side Effects

RunnableCommand is the generic logical command that is executed eagerly for its side effects.

RunnableCommand defines one abstract method run that computes a collection of Row records with the side effect, i.e. the result of executing a command.



run(sparkSession: SparkSession): Seq[Row]

run(sparkSession: SparkSession): Seq[Row]

Note	`RunnableCommand` logical operator is resolved to ExecutedCommandExec physical operator in BasicOperators execution planning strategy.

Note	`run` is executed when: `ExecutedCommandExec` executes logical RunnableCommand and caches the result as InternalRows `InsertIntoHadoopFsRelationCommand` is executed `QueryExecution` is requested to transform the result of executing DescribeTableCommand to a Hive-compatible output format

RunnableCommand Description

AddFileCommand

AddJarCommand

AlterDatabasePropertiesCommand

AlterTableAddPartitionCommand

AlterTableChangeColumnCommand

AlterTableDropPartitionCommand

AlterTableRecoverPartitionsCommand

AlterTableRenameCommand

AlterTableRenamePartitionCommand

AlterTableSerDePropertiesCommand

AlterTableSetLocationCommand

AlterTableSetPropertiesCommand

AlterTableUnsetPropertiesCommand

AlterViewAsCommand

AnalyzeColumnCommand

AnalyzePartitionCommand

AnalyzeTableCommand

CacheTableCommand

When executed, CacheTableCommand creates a DataFrame followed by registering a temporary view for the optional query.



CACHE LAZY? TABLE [table] (AS? [query])?

CACHE LAZY? TABLE [table] (AS? [query])?

CacheTableCommand requests the session-specific Catalog to cache the table.

Note	`CacheTableCommand` uses `SparkSession` to access the `Catalog`.

If the caching is not LAZY (which is not by default), CacheTableCommand creates a DataFrame for the table and counts the rows (that will trigger the caching).

Note	`CacheTableCommand` uses a Spark SQL pattern to trigger DataFrame caching by executing `count` operation.



val q = "CACHE TABLE ids AS SELECT * from range(5)"
scala> println(sql(q).queryExecution.logical.numberedTreeString)
00 CacheTableCommand `ids`, false
01    +- 'Project [*]
02       +- 'UnresolvedTableValuedFunction range, [5]

val q2 = "CACHE LAZY TABLE ids"
scala> println(sql(q2).queryExecution.logical.numberedTreeString)
17/05/17 06:16:39 WARN CacheManager: Asked to cache already cached data.
00 CacheTableCommand `ids`, true

val q = "CACHE TABLE ids AS SELECT * from range(5)"

scala> println(sql(q).queryExecution.logical.numberedTreeString)

00 CacheTableCommand `ids`, false

01 +- 'Project [*]

02 +- 'UnresolvedTableValuedFunction range, [5]

val q2 = "CACHE LAZY TABLE ids"

scala> println(sql(q2).queryExecution.logical.numberedTreeString)

17/05/17 06:16:39 WARN CacheManager: Asked to cache already cached data.

00 CacheTableCommand `ids`, true

ClearCacheCommand

CreateDatabaseCommand

CreateDataSourceTableAsSelectCommand

When executed, …FIXME

Used exclusively when DataSourceAnalysis posthoc logical resolution rule resolves a CreateTable logical operator with queries using non-Hive table providers (which is when DataFrameWriter saves a DataFrame to a non-Hive table or for Create Table As Select SQL statements)

CreateDataSourceTableCommand

CreateFunctionCommand

CreateHiveTableAsSelectCommand

CreateTableCommand

CreateTableLikeCommand

CreateTempViewUsing

CreateViewCommand

DescribeColumnCommand

DescribeDatabaseCommand

DescribeFunctionCommand

DescribeTableCommand

DropDatabaseCommand