Params and ParamMaps-spark技术分享

Params (and ParamMaps)

Params is the contract in Spark MLlib for ML components that take parameters.

Params has params collection of Param objects.



import org.apache.spark.ml.recommendation.ALS
val als = new ALS().
  setMaxIter(5).
  setRegParam(0.01).
  setUserCol("userId").
  setItemCol("movieId").
  setRatingCol("rating")
scala> :type als.params
Array[org.apache.spark.ml.param.Param[_]]

scala> println(als.explainParams)
alpha: alpha for implicit preference (default: 1.0)
checkpointInterval: set checkpoint interval (>= 1) or disable checkpoint (-1). E.g. 10 means that the cache will get checkpointed every 10 iterations (default: 10)
coldStartStrategy: strategy for dealing with unknown or new users/items at prediction time. This may be useful in cross-validation or production scenarios, for handling user/item ids the model has not seen in the training data. Supported values: nan,drop. (default: nan)
finalStorageLevel: StorageLevel for ALS model factors. (default: MEMORY_AND_DISK)
implicitPrefs: whether to use implicit preference (default: false)
intermediateStorageLevel: StorageLevel for intermediate datasets. Cannot be 'NONE'. (default: MEMORY_AND_DISK)
itemCol: column name for item ids. Ids must be within the integer value range. (default: item, current: movieId)
maxIter: maximum number of iterations (>= 0) (default: 10, current: 5)
nonnegative: whether to use nonnegative constraint for least squares (default: false)
numItemBlocks: number of item blocks (default: 10)
numUserBlocks: number of user blocks (default: 10)
predictionCol: prediction column name (default: prediction)
rank: rank of the factorization (default: 10)
ratingCol: column name for ratings (default: rating, current: rating)
regParam: regularization parameter (>= 0) (default: 0.1, current: 0.01)
seed: random seed (default: 1994790107)
userCol: column name for user ids. Ids must be within the integer value range. (default: user, current: userId)

import org.apache.spark.ml.recommendation.ALS

val als = new ALS().

setMaxIter(5).

setRegParam(0.01).

setUserCol("userId").

setItemCol("movieId").

setRatingCol("rating")

scala> :type als.params

Array[org.apache.spark.ml.param.Param[_]]

scala> println(als.explainParams)

alpha: alpha for implicit preference (default: 1.0)

checkpointInterval: set checkpoint interval (>= 1) or disable checkpoint (-1). E.g. 10 means that the cache will get checkpointed every 10 iterations (default: 10)

coldStartStrategy: strategy for dealing with unknown or new users/items at prediction time. This may be useful in cross-validation or production scenarios, for handling user/item ids the model has not seen in the training data. Supported values: nan,drop. (default: nan)

finalStorageLevel: StorageLevel for ALS model factors. (default: MEMORY_AND_DISK)

implicitPrefs: whether to use implicit preference (default: false)

intermediateStorageLevel: StorageLevel for intermediate datasets. Cannot be 'NONE'. (default: MEMORY_AND_DISK)

itemCol: column name for item ids. Ids must be within the integer value range. (default: item, current: movieId)

maxIter: maximum number of iterations (>= 0) (default: 10, current: 5)

nonnegative: whether to use nonnegative constraint for least squares (default: false)

numItemBlocks: number of item blocks (default: 10)

numUserBlocks: number of user blocks (default: 10)

predictionCol: prediction column name (default: prediction)

rank: rank of the factorization (default: 10)

ratingCol: column name for ratings (default: rating, current: rating)

regParam: regularization parameter (>= 0) (default: 0.1, current: 0.01)

seed: random seed (default: 1994790107)

userCol: column name for user ids. Ids must be within the integer value range. (default: user, current: userId)



import org.apache.spark.ml.tuning.CrossValidator
val cv = new CrossValidator
scala> println(cv.explainParams)
estimator: estimator for selection (undefined)
estimatorParamMaps: param maps for the estimator (undefined)
evaluator: evaluator used to select hyper-parameters that maximize the validated metric (undefined)
numFolds: number of folds for cross validation (>= 2) (default: 3)
seed: random seed (default: -1191137437)

import org.apache.spark.ml.tuning.CrossValidator

val cv = new CrossValidator

scala> println(cv.explainParams)

estimator: estimator for selection (undefined)

estimatorParamMaps: param maps for the estimator (undefined)

evaluator: evaluator used to select hyper-parameters that maximize the validated metric (undefined)

numFolds: number of folds for cross validation (>= 2) (default: 3)

seed: random seed (default: -1191137437)

Params comes with $ (dollar) method for Spark MLlib developers to access the user-defined or the default value of a parameter.

Params Contract



package org.apache.spark.ml.param

trait Params {
  def copy(extra: ParamMap): Params
}

package org.apache.spark.ml.param

trait Params {

def copy(extra: ParamMap): Params

}

Table 1. (Subset of) Params Contract
Method	Description
`copy`

Explaining Parameters — `explainParams` Method



explainParams(): String

explainParams(): String

explainParams takes params collection of parameters and converts every parameter to a corresponding help text with the param name, the description and optionally the default and the user-defined values if available.



import org.apache.spark.ml.recommendation.ALS
val als = new ALS().
  setMaxIter(5).
  setRegParam(0.01).
  setUserCol("userId").
  setItemCol("movieId").
  setRatingCol("rating")
scala> println(als.explainParams)
alpha: alpha for implicit preference (default: 1.0)
checkpointInterval: set checkpoint interval (>= 1) or disable checkpoint (-1). E.g. 10 means that the cache will get checkpointed every 10 iterations (default: 10)
coldStartStrategy: strategy for dealing with unknown or new users/items at prediction time. This may be useful in cross-validation or production scenarios, for handling user/item ids the model has not seen in the training data. Supported values: nan,drop. (default: nan)
finalStorageLevel: StorageLevel for ALS model factors. (default: MEMORY_AND_DISK)
implicitPrefs: whether to use implicit preference (default: false)
intermediateStorageLevel: StorageLevel for intermediate datasets. Cannot be 'NONE'. (default: MEMORY_AND_DISK)
itemCol: column name for item ids. Ids must be within the integer value range. (default: item, current: movieId)
maxIter: maximum number of iterations (>= 0) (default: 10, current: 5)
nonnegative: whether to use nonnegative constraint for least squares (default: false)
numItemBlocks: number of item blocks (default: 10)
numUserBlocks: number of user blocks (default: 10)
predictionCol: prediction column name (default: prediction)
rank: rank of the factorization (default: 10)
ratingCol: column name for ratings (default: rating, current: rating)
regParam: regularization parameter (>= 0) (default: 0.1, current: 0.01)
seed: random seed (default: 1994790107)
userCol: column name for user ids. Ids must be within the integer value range. (default: user, current: userId)