spark技术分享HiveUtils
HiveUtils is used to create a HiveClientImpl that HiveExternalCatalog uses to interact with a Hive metastore.
|
Tip
|
Enable Add the following line to
Refer to Logging. |
Creating HiveClientImpl — newClientForMetadata Method
|
1 2 3 4 5 6 7 8 9 10 11 |
newClientForMetadata( conf: SparkConf, hadoopConf: Configuration): HiveClient (1) newClientForMetadata( conf: SparkConf, hadoopConf: Configuration, configurations: Map[String, String]): HiveClient |
-
Executes the other
newClientForMetadatawith time configurations formatted
Internally, newClientForMetadata creates a new SQLConf with spark.sql properties only (from the input SparkConf).
newClientForMetadata then creates a IsolatedClientLoader per the input parameters and the following configuration properties:
You should see one of the following INFO messages in the logs:
|
1 2 3 4 5 6 7 |
Initializing HiveMetastoreConnection version [hiveMetastoreVersion] using Spark classes. Initializing HiveMetastoreConnection version [hiveMetastoreVersion] using maven. Initializing HiveMetastoreConnection version [hiveMetastoreVersion] using [jars] |
In the end, newClientForMetadata requests the IsolatedClientLoader to create a HiveClientImpl.
|
Note
|
newClientForMetadata is used exclusively when HiveExternalCatalog is requested for a HiveClient.
|
