org.apache.spark.mllib.feature

Class IDF

java.lang.Object

org.apache.spark.mllib.feature.IDF

public class IDF
extends java.lang.Object

Inverse document frequency (IDF). The standard formulation is used: idf = log((m + 1) / (d(t) + 1)), where m is the total number of documents and d(t) is the number of documents that contain term t.

This implementation supports filtering out terms which do not appear in a minimum number of documents (controlled by the variable minDocFreq). For terms that are not in at least minDocFreq documents, the IDF is found as 0, resulting in TF-IDFs of 0.

param: minDocFreq minimum of documents in which a term should appear for filtering

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	IDF.DocumentFrequencyAggregator Document frequency aggregator.

Constructor Summary

Constructors

Constructor and Description
IDF()
IDF(int minDocFreq)

Method Summary

Methods

Modifier and Type	Method and Description
IDFModel	fit(JavaRDD<Vector> dataset) Computes the inverse document frequency.
IDFModel	fit(RDD<Vector> dataset) Computes the inverse document frequency.
int	minDocFreq()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

IDF

public IDF(int minDocFreq)

IDF

public IDF()

Method Detail

minDocFreq

public int minDocFreq()

fit

public IDFModel fit(RDD<Vector> dataset)

Computes the inverse document frequency.

Parameters:

dataset - an RDD of term frequency vectors

Returns:

(undocumented)

fit

public IDFModel fit(JavaRDD<Vector> dataset)

Computes the inverse document frequency.

Parameters:

dataset - a JavaRDD of term frequency vectors

Returns:

(undocumented)