JavaRDD

final def !=(arg0: AnyRef): Boolean

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: AnyRef): Boolean

final def ==(arg0: Any): Boolean

def aggregate[U](zeroValue: U)(seqOp: Function2[U, T, U], combOp: Function2[U, U, U]): U

Aggregate the elements of each partition, and then the results for all the partitions, using given combine functions and a neutral "zero value".

final def asInstanceOf[T0]: T0

def cache(): JavaRDD[T]

Persist this RDD with the default storage level (MEMORY_ONLY).

def cartesian[U](other: JavaRDDLike[U, _]): JavaPairRDD[T, U]

Return the Cartesian product of this RDD and another one, that is, the RDD of all pairs of elements (a, b) where a is in this and b is in other.

def checkpoint(): Unit

Mark this RDD for checkpointing.

implicit val classTag: ClassTag[T]

def clone(): AnyRef

def coalesce(numPartitions: Int, shuffle: Boolean): JavaRDD[T]

Return a new RDD that is reduced into numPartitions partitions.

def coalesce(numPartitions: Int): JavaRDD[T]

Return a new RDD that is reduced into numPartitions partitions.

def collect(): List[T]

Return an array that contains all of the elements in this RDD.

def collectAsync(): JavaFutureAction[List[T]]

The asynchronous version of collect, which returns a future for retrieving an array containing all of the elements in this RDD.

def collectPartitions(partitionIds: Array[Int]): Array[List[T]]

Return an array that contains all of the elements in a specific partition of this RDD.

def context: SparkContext

The org.apache.spark.SparkContext that this RDD was created on.

def count(): Long

Return the number of elements in the RDD.

def countApprox(timeout: Long): PartialResult[BoundedDouble]

:: Experimental :: Approximate version of count() that returns a potentially incomplete result within a timeout, even if not all tasks have finished.

def countApprox(timeout: Long, confidence: Double): PartialResult[BoundedDouble]

:: Experimental :: Approximate version of count() that returns a potentially incomplete result within a timeout, even if not all tasks have finished.

def countApproxDistinct(relativeSD: Double): Long

Return approximate number of distinct elements in the RDD.

def countAsync(): JavaFutureAction[Long]

The asynchronous version of count, which returns a future for counting the number of elements in this RDD.

def countByValue(): Map[T, Long]

Return the count of each unique value in this RDD as a map of (value, count) pairs.

def countByValueApprox(timeout: Long): PartialResult[Map[T, BoundedDouble]]

(Experimental) Approximate version of countByValue().

def countByValueApprox(timeout: Long, confidence: Double): PartialResult[Map[T, BoundedDouble]]

(Experimental) Approximate version of countByValue().

def distinct(numPartitions: Int): JavaRDD[T]

Return a new RDD containing the distinct elements in this RDD.

def distinct(): JavaRDD[T]

Return a new RDD containing the distinct elements in this RDD.

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def filter(f: Function[T, Boolean]): JavaRDD[T]

Return a new RDD containing only the elements that satisfy a predicate.

def finalize(): Unit

def first(): T

Return the first element in this RDD.

def flatMap[U](f: FlatMapFunction[T, U]): JavaRDD[U]

Return a new RDD by first applying a function to all elements of this RDD, and then flattening the results.

def flatMapToDouble(f: DoubleFlatMapFunction[T]): JavaDoubleRDD

Return a new RDD by first applying a function to all elements of this RDD, and then flattening the results.

def flatMapToPair[K2, V2](f: PairFlatMapFunction[T, K2, V2]): JavaPairRDD[K2, V2]

Return a new RDD by first applying a function to all elements of this RDD, and then flattening the results.

def fold(zeroValue: T)(f: Function2[T, T, T]): T

Aggregate the elements of each partition, and then the results for all the partitions, using a given associative and commutative function and a neutral "zero value".

def foreach(f: VoidFunction[T]): Unit

Applies a function f to all elements of this RDD.

def foreachAsync(f: VoidFunction[T]): JavaFutureAction[Void]

The asynchronous version of the foreach action, which applies a function f to all the elements of this RDD.

def foreachPartition(f: VoidFunction[Iterator[T]]): Unit

Applies a function f to each partition of this RDD.

def foreachPartitionAsync(f: VoidFunction[Iterator[T]]): JavaFutureAction[Void]

The asynchronous version of the foreachPartition action, which applies a function f to each partition of this RDD.

def getCheckpointFile(): Optional[String]

Gets the name of the file to which this RDD was checkpointed

final def getClass(): Class[_]

def getStorageLevel: StorageLevel

Get the RDD's current storage level, or StorageLevel.

def glom(): JavaRDD[List[T]]

Return an RDD created by coalescing all elements within each partition into an array.

def groupBy[U](f: Function[T, U], numPartitions: Int): JavaPairRDD[U, Iterable[T]]

Return an RDD of grouped elements.

def groupBy[U](f: Function[T, U]): JavaPairRDD[U, Iterable[T]]

Return an RDD of grouped elements.

def hashCode(): Int

def id: Int

A unique ID for this RDD (within its SparkContext).

def intersection(other: JavaRDD[T]): JavaRDD[T]

Return the intersection of this RDD and another one.

def isCheckpointed: Boolean

Return whether this RDD has been checkpointed or not

def isEmpty(): Boolean

final def isInstanceOf[T0]: Boolean

def iterator(split: Partition, taskContext: TaskContext): Iterator[T]

Internal method to this RDD; will read from cache if applicable, or otherwise compute it.

def keyBy[U](f: Function[T, U]): JavaPairRDD[U, T]

Creates tuples of the elements in this RDD by applying f.

def map[R](f: Function[T, R]): JavaRDD[R]

Return a new RDD by applying a function to all elements of this RDD.

def mapPartitions[U](f: FlatMapFunction[Iterator[T], U], preservesPartitioning: Boolean): JavaRDD[U]

Return a new RDD by applying a function to each partition of this RDD.

def mapPartitions[U](f: FlatMapFunction[Iterator[T], U]): JavaRDD[U]

Return a new RDD by applying a function to each partition of this RDD.

def mapPartitionsToDouble(f: DoubleFlatMapFunction[Iterator[T]], preservesPartitioning: Boolean): JavaDoubleRDD

Return a new RDD by applying a function to each partition of this RDD.

def mapPartitionsToDouble(f: DoubleFlatMapFunction[Iterator[T]]): JavaDoubleRDD

Return a new RDD by applying a function to each partition of this RDD.

def mapPartitionsToPair[K2, V2](f: PairFlatMapFunction[Iterator[T], K2, V2], preservesPartitioning: Boolean): JavaPairRDD[K2, V2]

Return a new RDD by applying a function to each partition of this RDD.

def mapPartitionsToPair[K2, V2](f: PairFlatMapFunction[Iterator[T], K2, V2]): JavaPairRDD[K2, V2]

Return a new RDD by applying a function to each partition of this RDD.

def mapPartitionsWithIndex[R](f: Function2[Integer, Iterator[T], Iterator[R]], preservesPartitioning: Boolean = false): JavaRDD[R]

Return a new RDD by applying a function to each partition of this RDD, while tracking the index of the original partition.

def mapToDouble[R](f: DoubleFunction[T]): JavaDoubleRDD

Return a new RDD by applying a function to all elements of this RDD.

def mapToPair[K2, V2](f: PairFunction[T, K2, V2]): JavaPairRDD[K2, V2]

Return a new RDD by applying a function to all elements of this RDD.

def max(comp: Comparator[T]): T

Returns the maximum element from this RDD as defined by the specified Comparator[T].

def min(comp: Comparator[T]): T

Returns the minimum element from this RDD as defined by the specified Comparator[T].

def name(): String

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

def partitioner: Optional[Partitioner]

The partitioner of this RDD.

def partitions: List[Partition]

Set of partitions in this RDD.

def persist(newLevel: StorageLevel): JavaRDD[T]

Set this RDD's storage level to persist its values across operations after the first time it is computed.

def pipe(command: List[String], env: Map[String, String]): JavaRDD[String]

Return an RDD created by piping elements to a forked external process.

def pipe(command: List[String]): JavaRDD[String]

Return an RDD created by piping elements to a forked external process.

def pipe(command: String): JavaRDD[String]

Return an RDD created by piping elements to a forked external process.

def randomSplit(weights: Array[Double], seed: Long): Array[JavaRDD[T]]

Randomly splits this RDD with the provided weights.

def randomSplit(weights: Array[Double]): Array[JavaRDD[T]]

Randomly splits this RDD with the provided weights.

val rdd: RDD[T]

def reduce(f: Function2[T, T, T]): T

Reduces the elements of this RDD using the specified commutative and associative binary operator.

def repartition(numPartitions: Int): JavaRDD[T]

Return a new RDD that has exactly numPartitions partitions.

def sample(withReplacement: Boolean, fraction: Double, seed: Long): JavaRDD[T]

Return a sampled subset of this RDD.

def sample(withReplacement: Boolean, fraction: Double): JavaRDD[T]

Return a sampled subset of this RDD.

def saveAsObjectFile(path: String): Unit

Save this RDD as a SequenceFile of serialized objects.

def saveAsTextFile(path: String, codec: Class[_ <: CompressionCodec]): Unit

Save this RDD as a compressed text file, using string representations of elements.

def saveAsTextFile(path: String): Unit

Save this RDD as a text file, using string representations of elements.

def setName(name: String): JavaRDD[T]

Assign a name to this RDD

def sortBy[S](f: Function[T, S], ascending: Boolean, numPartitions: Int): JavaRDD[T]

Return this RDD sorted by the given key function.

def subtract(other: JavaRDD[T], p: Partitioner): JavaRDD[T]

Return an RDD with the elements from this that are not in other.

def subtract(other: JavaRDD[T], numPartitions: Int): JavaRDD[T]

Return an RDD with the elements from this that are not in other.

def subtract(other: JavaRDD[T]): JavaRDD[T]

Return an RDD with the elements from this that are not in other.

final def synchronized[T0](arg0: ⇒ T0): T0

def take(num: Int): List[T]

Take the first num elements of the RDD.

def takeAsync(num: Int): JavaFutureAction[List[T]]

The asynchronous version of the take action, which returns a future for retrieving the first num elements of this RDD.

def takeOrdered(num: Int): List[T]

Returns the first k (smallest) elements from this RDD using the natural ordering for T while maintain the order.

def takeOrdered(num: Int, comp: Comparator[T]): List[T]

Returns the first k (smallest) elements from this RDD as defined by the specified Comparator[T] and maintains the order.

def takeSample(withReplacement: Boolean, num: Int, seed: Long): List[T]

def takeSample(withReplacement: Boolean, num: Int): List[T]

def toDebugString(): String

A description of this RDD and its recursive dependencies for debugging.

def toLocalIterator(): Iterator[T]

Return an iterator that contains all of the elements in this RDD.

def toString(): String

def top(num: Int): List[T]

Returns the top k (largest) elements from this RDD using the natural ordering for T.

def top(num: Int, comp: Comparator[T]): List[T]

Returns the top k (largest) elements from this RDD as defined by the specified Comparator[T].

def treeAggregate[U](zeroValue: U, seqOp: Function2[U, T, U], combOp: Function2[U, U, U]): U

org.apache.spark.api.java.JavaRDDLike#treeAggregate with suggested depth 2.

def treeAggregate[U](zeroValue: U, seqOp: Function2[U, T, U], combOp: Function2[U, U, U], depth: Int): U

Aggregates the elements of this RDD in a multi-level tree pattern.

def treeReduce(f: Function2[T, T, T]): T

org.apache.spark.api.java.JavaRDDLike#treeReduce with suggested depth 2.

def treeReduce(f: Function2[T, T, T], depth: Int): T

Reduces the elements of this RDD in a multi-level tree pattern.

def union(other: JavaRDD[T]): JavaRDD[T]

Return the union of this RDD and another one.

def unpersist(blocking: Boolean): JavaRDD[T]

Mark the RDD as non-persistent, and remove all blocks for it from memory and disk.

def unpersist(): JavaRDD[T]

Mark the RDD as non-persistent, and remove all blocks for it from memory and disk.

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

def wrapRDD(rdd: RDD[T]): JavaRDD[T]

def zip[U](other: JavaRDDLike[U, _]): JavaPairRDD[T, U]

Zips this RDD with another one, returning key-value pairs with the first element in each RDD, second element in each RDD, etc.

def zipPartitions[U, V](other: JavaRDDLike[U, _], f: FlatMapFunction2[Iterator[T], Iterator[U], V]): JavaRDD[V]

Zip this RDD's partitions with one (or more) RDD(s) and return a new RDD by applying a function to the zipped partitions.

def zipWithIndex(): JavaPairRDD[T, Long]

Zips this RDD with its element indices.

def zipWithUniqueId(): JavaPairRDD[T, Long]

Zips this RDD with generated unique Long ids.

def splits: List[Partition]

def toArray(): List[T]

Return an array that contains all of the elements in this RDD.