StatFunctions Helper Object-spark技术分享

StatFunctions Helper Object

StatFunctions is a Scala object that defines the methods that are used for…FIXME

Table 1. StatFunctions API

Method

Description



calculateCov(df: DataFrame, cols: Seq[String]): Double

calculateCov(df: DataFrame, cols: Seq[String]): Double

crossTabulate



crossTabulate(df: DataFrame, col1: String, col2: String): DataFrame

crossTabulate(df: DataFrame, col1: String, col2: String): DataFrame

multipleApproxQuantiles



multipleApproxQuantiles(
  df: DataFrame,
  cols: Seq[String],
  probabilities: Seq[Double],
  relativeError: Double): Seq[Seq[Double]]

multipleApproxQuantiles(

df: DataFrame,

cols: Seq[String],

probabilities: Seq[Double],

relativeError: Double): Seq[Seq[Double]]

pearsonCorrelation



pearsonCorrelation(df: DataFrame, cols: Seq[String]): Double

pearsonCorrelation(df: DataFrame, cols: Seq[String]): Double

summary



summary(ds: Dataset[_], statistics: Seq[String]): DataFrame

summary(ds: Dataset[_], statistics: Seq[String]): DataFrame

`calculateCov` Method



calculateCov(df: DataFrame, cols: Seq[String]): Double

calculateCov(df: DataFrame, cols: Seq[String]): Double

calculateCov…FIXME

Note	`calculateCov` is used when…FIXME

`crossTabulate` Method



crossTabulate(df: DataFrame, col1: String, col2: String): DataFrame

crossTabulate(df: DataFrame, col1: String, col2: String): DataFrame

crossTabulate…FIXME

Note	`crossTabulate` is used when…FIXME

`multipleApproxQuantiles` Method



multipleApproxQuantiles(
  df: DataFrame,
  cols: Seq[String],
  probabilities: Seq[Double],
  relativeError: Double): Seq[Seq[Double]]

multipleApproxQuantiles(

df: DataFrame,

cols: Seq[String],

probabilities: Seq[Double],

relativeError: Double): Seq[Seq[Double]]

multipleApproxQuantiles…FIXME

Note	`multipleApproxQuantiles` is used when…FIXME

`pearsonCorrelation` Method



pearsonCorrelation(df: DataFrame, cols: Seq[String]): Double

pearsonCorrelation(df: DataFrame, cols: Seq[String]): Double

pearsonCorrelation…FIXME

Note	`pearsonCorrelation` is used when…FIXME

Calculating Statistics For Dataset — `summary` Method



summary(ds: Dataset[_], statistics: Seq[String]): DataFrame

summary(ds: Dataset[_], statistics: Seq[String]): DataFrame

summary…FIXME

Note	`summary` is used exclusively when Dataset.summary action is used.

StatFunctions Helper Object