DataFrameStatFunctions — Working With Statistic Functions-spark技术分享

DataFrameStatFunctions — Working With Statistic Functions

DataFrameStatFunctions is used to work with statistic functions in a structured query (a DataFrame).

Table 1. DataFrameStatFunctions API

Method

Description



approxQuantile(
  cols: Array[String],
  probabilities: Array[Double],
  relativeError: Double): Array[Array[Double]]
approxQuantile(
  col: String,
  probabilities: Array[Double],
  relativeError: Double): Array[Double]

approxQuantile(

cols: Array[String],

probabilities: Array[Double],

relativeError: Double): Array[Array[Double]]

approxQuantile(

col: String,

probabilities: Array[Double],

relativeError: Double): Array[Double]

bloomFilter



bloomFilter(col: Column, expectedNumItems: Long, fpp: Double): BloomFilter
bloomFilter(col: Column, expectedNumItems: Long, numBits: Long): BloomFilter
bloomFilter(colName: String, expectedNumItems: Long, fpp: Double): BloomFilter
bloomFilter(colName: String, expectedNumItems: Long, numBits: Long): BloomFilter

bloomFilter(col: Column, expectedNumItems: Long, fpp: Double): BloomFilter

bloomFilter(col: Column, expectedNumItems: Long, numBits: Long): BloomFilter

bloomFilter(colName: String, expectedNumItems: Long, fpp: Double): BloomFilter

bloomFilter(colName: String, expectedNumItems: Long, numBits: Long): BloomFilter

corr



corr(col1: String, col2: String): Double
corr(col1: String, col2: String, method: String): Double

corr(col1: String, col2: String): Double

corr(col1: String, col2: String, method: String): Double

countMinSketch



countMinSketch(col: Column, eps: Double, confidence: Double, seed: Int): CountMinSketch
countMinSketch(col: Column, depth: Int, width: Int, seed: Int): CountMinSketch
countMinSketch(colName: String, eps: Double, confidence: Double, seed: Int): CountMinSketch
countMinSketch(colName: String, depth: Int, width: Int, seed: Int): CountMinSketch

countMinSketch(col: Column, eps: Double, confidence: Double, seed: Int): CountMinSketch

countMinSketch(col: Column, depth: Int, width: Int, seed: Int): CountMinSketch

countMinSketch(colName: String, eps: Double, confidence: Double, seed: Int): CountMinSketch

countMinSketch(colName: String, depth: Int, width: Int, seed: Int): CountMinSketch

cov



cov(col1: String, col2: String): Double

cov(col1: String, col2: String): Double

crosstab



crosstab(col1: String, col2: String): DataFrame

crosstab(col1: String, col2: String): DataFrame

freqItems



freqItems(cols: Array[String]): DataFrame
freqItems(cols: Array[String], support: Double): DataFrame
freqItems(cols: Seq[String]): DataFrame
freqItems(cols: Seq[String], support: Double): DataFrame

freqItems(cols: Array[String]): DataFrame

freqItems(cols: Array[String], support: Double): DataFrame

freqItems(cols: Seq[String]): DataFrame

freqItems(cols: Seq[String], support: Double): DataFrame

sampleBy



sampleBy[T](col: String, fractions: Map[T, Double], seed: Long): DataFrame

sampleBy[T](col: String, fractions: Map[T, Double], seed: Long): DataFrame

DataFrameStatFunctions is available using stat untyped transformation.



val q: DataFrame = ...
q.stat

val q: DataFrame = ...

q.stat

`approxQuantile` Method



approxQuantile(
  cols: Array[String],
  probabilities: Array[Double],
  relativeError: Double): Array[Array[Double]]
approxQuantile(
  col: String,
  probabilities: Array[Double],
  relativeError: Double): Array[Double]

approxQuantile(

cols: Array[String],

probabilities: Array[Double],

relativeError: Double): Array[Array[Double]]

approxQuantile(

col: String,

probabilities: Array[Double],

relativeError: Double): Array[Double]

approxQuantile…FIXME

`bloomFilter` Method



bloomFilter(col: Column, expectedNumItems: Long, fpp: Double): BloomFilter
bloomFilter(col: Column, expectedNumItems: Long, numBits: Long): BloomFilter
bloomFilter(colName: String, expectedNumItems: Long, fpp: Double): BloomFilter
bloomFilter(colName: String, expectedNumItems: Long, numBits: Long): BloomFilter

bloomFilter(col: Column, expectedNumItems: Long, fpp: Double): BloomFilter

bloomFilter(col: Column, expectedNumItems: Long, numBits: Long): BloomFilter

bloomFilter(colName: String, expectedNumItems: Long, fpp: Double): BloomFilter

bloomFilter(colName: String, expectedNumItems: Long, numBits: Long): BloomFilter

bloomFilter…FIXME

`buildBloomFilter` Internal Method



buildBloomFilter(col: Column, zero: BloomFilter): BloomFilter

buildBloomFilter(col: Column, zero: BloomFilter): BloomFilter

buildBloomFilter…FIXME

Note	`convertToDouble` is used when…FIXME

`corr` Method



corr(col1: String, col2: String): Double
corr(col1: String, col2: String, method: String): Double

corr(col1: String, col2: String): Double

corr(col1: String, col2: String, method: String): Double

corr…FIXME

`countMinSketch` Method



countMinSketch(col: Column, eps: Double, confidence: Double, seed: Int): CountMinSketch
countMinSketch(col: Column, depth: Int, width: Int, seed: Int): CountMinSketch
countMinSketch(colName: String, eps: Double, confidence: Double, seed: Int): CountMinSketch
countMinSketch(colName: String, depth: Int, width: Int, seed: Int): CountMinSketch
// PRIVATE API
countMinSketch(col: Column, zero: CountMinSketch): CountMinSketch

countMinSketch(col: Column, eps: Double, confidence: Double, seed: Int): CountMinSketch

countMinSketch(col: Column, depth: Int, width: Int, seed: Int): CountMinSketch

countMinSketch(colName: String, eps: Double, confidence: Double, seed: Int): CountMinSketch

countMinSketch(colName: String, depth: Int, width: Int, seed: Int): CountMinSketch

// PRIVATE API

countMinSketch(col: Column, zero: CountMinSketch): CountMinSketch

countMinSketch…FIXME

`cov` Method



cov(col1: String, col2: String): Double

cov(col1: String, col2: String): Double

cov…FIXME

`crosstab` Method



crosstab(col1: String, col2: String): DataFrame

crosstab(col1: String, col2: String): DataFrame

crosstab…FIXME

`freqItems` Method



freqItems(cols: Array[String]): DataFrame
freqItems(cols: Array[String], support: Double): DataFrame
freqItems(cols: Seq[String]): DataFrame
freqItems(cols: Seq[String], support: Double): DataFrame

freqItems(cols: Array[String]): DataFrame

freqItems(cols: Array[String], support: Double): DataFrame

freqItems(cols: Seq[String]): DataFrame

freqItems(cols: Seq[String], support: Double): DataFrame

freqItems…FIXME

`sampleBy` Method



sampleBy[T](col: String, fractions: Map[T, Double], seed: Long): DataFrame

sampleBy[T](col: String, fractions: Map[T, Double], seed: Long): DataFrame

sampleBy…FIXME

DataFrameStatFunctions — Working With Statistic Functions