KafkaUtils

Value Members

1. final def !=(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

2. final def !=(arg0: Any): Boolean

   Definition Classes

   Any

3. final def ##(): Int

   Definition Classes

   AnyRef → Any

4. final def ==(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

5. final def ==(arg0: Any): Boolean

   Definition Classes

   Any

6. final def asInstanceOf[T0]: T0

   Definition Classes

   Any

7. def clone(): AnyRef

   Attributes

   protected[java.lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... )

8. def createDirectStream[K, V, KD <: Decoder[K], VD <: Decoder[V]](jssc: JavaStreamingContext, keyClass: Class[K], valueClass: Class[V], keyDecoderClass: Class[KD], valueDecoderClass: Class[VD], kafkaParams: Map[String, String], topics: Set[String]): JavaPairInputDStream[K, V]

   :: Experimental :: Create an input stream that directly pulls messages from Kafka Brokers without using any receiver.

   :: Experimental :: Create an input stream that directly pulls messages from Kafka Brokers without using any receiver. This stream can guarantee that each message from Kafka is included in transformations exactly once (see points below).

   Points to note:

   • No receivers: This stream does not use any receiver. It directly queries Kafka
   • Offsets: This does not use Zookeeper to store offsets. The consumed offsets are tracked by the stream itself. For interoperability with Kafka monitoring tools that depend on Zookeeper, you have to update Kafka/Zookeeper yourself from the streaming application. You can access the offsets used in each batch from the generated RDDs (see org.apache.spark.streaming.kafka.HasOffsetRanges).
   • Failure Recovery: To recover from driver failures, you have to enable checkpointing in the StreamingContext. The information on consumed offset can be recovered from the checkpoint. See the programming guide for details (constraints, etc.).
   • End-to-end semantics: This stream ensures that every records is effectively received and transformed exactly once, but gives no guarantees on whether the transformed data are outputted exactly once. For end-to-end exactly-once semantics, you have to either ensure that the output operation is idempotent, or use transactions to output records atomically. See the programming guide for more details.

   jssc

   JavaStreamingContext object

   keyClass

   Class of the keys in the Kafka records

   valueClass

   Class of the values in the Kafka records

   keyDecoderClass

   Class of the key decoder

   valueDecoderClass

   Class type of the value decoder

   kafkaParams

   Kafka configuration parameters. Requires "metadata.broker.list" or "bootstrap.servers" to be set with Kafka broker(s) (NOT zookeeper servers), specified in host1:port1,host2:port2 form. If not starting from a checkpoint, "auto.offset.reset" may be set to "largest" or "smallest" to determine where the stream starts (defaults to "largest")

   topics

   Names of the topics to consume

   Annotations

   @Experimental()

9. def createDirectStream[K, V, KD <: Decoder[K], VD <: Decoder[V], R](jssc: JavaStreamingContext, keyClass: Class[K], valueClass: Class[V], keyDecoderClass: Class[KD], valueDecoderClass: Class[VD], recordClass: Class[R], kafkaParams: Map[String, String], fromOffsets: Map[TopicAndPartition, Long], messageHandler: Function[MessageAndMetadata[K, V], R]): JavaInputDStream[R]

   :: Experimental :: Create an input stream that directly pulls messages from Kafka Brokers without using any receiver.

   :: Experimental :: Create an input stream that directly pulls messages from Kafka Brokers without using any receiver. This stream can guarantee that each message from Kafka is included in transformations exactly once (see points below).

   Points to note:

   • No receivers: This stream does not use any receiver. It directly queries Kafka
   • Offsets: This does not use Zookeeper to store offsets. The consumed offsets are tracked by the stream itself. For interoperability with Kafka monitoring tools that depend on Zookeeper, you have to update Kafka/Zookeeper yourself from the streaming application. You can access the offsets used in each batch from the generated RDDs (see org.apache.spark.streaming.kafka.HasOffsetRanges).
   • Failure Recovery: To recover from driver failures, you have to enable checkpointing in the StreamingContext. The information on consumed offset can be recovered from the checkpoint. See the programming guide for details (constraints, etc.).
   • End-to-end semantics: This stream ensures that every records is effectively received and transformed exactly once, but gives no guarantees on whether the transformed data are outputted exactly once. For end-to-end exactly-once semantics, you have to either ensure that the output operation is idempotent, or use transactions to output records atomically. See the programming guide for more details.

   jssc

   JavaStreamingContext object

   keyClass

   Class of the keys in the Kafka records

   valueClass

   Class of the values in the Kafka records

   keyDecoderClass

   Class of the key decoder

   valueDecoderClass

   Class of the value decoder

   recordClass

   Class of the records in DStream

   kafkaParams

   Kafka configuration parameters. Requires "metadata.broker.list" or "bootstrap.servers" to be set with Kafka broker(s) (NOT zookeeper servers), specified in host1:port1,host2:port2 form.

   fromOffsets

   Per-topic/partition Kafka offsets defining the (inclusive) starting point of the stream

   messageHandler

   Function for translating each message and metadata into the desired type

   Annotations

   @Experimental()

10. def createDirectStream[K, V, KD <: Decoder[K], VD <: Decoder[V]](ssc: StreamingContext, kafkaParams: Map[String, String], topics: Set[String])(implicit arg0: ClassTag[K], arg1: ClassTag[V], arg2: ClassTag[KD], arg3: ClassTag[VD]): InputDStream[(K, V)]

    :: Experimental :: Create an input stream that directly pulls messages from Kafka Brokers without using any receiver.

    :: Experimental :: Create an input stream that directly pulls messages from Kafka Brokers without using any receiver. This stream can guarantee that each message from Kafka is included in transformations exactly once (see points below).

    Points to note:

    • No receivers: This stream does not use any receiver. It directly queries Kafka
    • Offsets: This does not use Zookeeper to store offsets. The consumed offsets are tracked by the stream itself. For interoperability with Kafka monitoring tools that depend on Zookeeper, you have to update Kafka/Zookeeper yourself from the streaming application. You can access the offsets used in each batch from the generated RDDs (see org.apache.spark.streaming.kafka.HasOffsetRanges).
    • Failure Recovery: To recover from driver failures, you have to enable checkpointing in the StreamingContext. The information on consumed offset can be recovered from the checkpoint. See the programming guide for details (constraints, etc.).
    • End-to-end semantics: This stream ensures that every records is effectively received and transformed exactly once, but gives no guarantees on whether the transformed data are outputted exactly once. For end-to-end exactly-once semantics, you have to either ensure that the output operation is idempotent, or use transactions to output records atomically. See the programming guide for more details.

    ssc

    StreamingContext object

    kafkaParams

    Kafka configuration parameters. Requires "metadata.broker.list" or "bootstrap.servers" to be set with Kafka broker(s) (NOT zookeeper servers), specified in host1:port1,host2:port2 form. If not starting from a checkpoint, "auto.offset.reset" may be set to "largest" or "smallest" to determine where the stream starts (defaults to "largest")

    topics

    Names of the topics to consume

    Annotations

    @Experimental()

11. def createDirectStream[K, V, KD <: Decoder[K], VD <: Decoder[V], R](ssc: StreamingContext, kafkaParams: Map[String, String], fromOffsets: Map[TopicAndPartition, Long], messageHandler: (MessageAndMetadata[K, V]) ⇒ R)(implicit arg0: ClassTag[K], arg1: ClassTag[V], arg2: ClassTag[KD], arg3: ClassTag[VD], arg4: ClassTag[R]): InputDStream[R]

    :: Experimental :: Create an input stream that directly pulls messages from Kafka Brokers without using any receiver.

    :: Experimental :: Create an input stream that directly pulls messages from Kafka Brokers without using any receiver. This stream can guarantee that each message from Kafka is included in transformations exactly once (see points below).

    Points to note:

    • No receivers: This stream does not use any receiver. It directly queries Kafka
    • Offsets: This does not use Zookeeper to store offsets. The consumed offsets are tracked by the stream itself. For interoperability with Kafka monitoring tools that depend on Zookeeper, you have to update Kafka/Zookeeper yourself from the streaming application. You can access the offsets used in each batch from the generated RDDs (see org.apache.spark.streaming.kafka.HasOffsetRanges).
    • Failure Recovery: To recover from driver failures, you have to enable checkpointing in the StreamingContext. The information on consumed offset can be recovered from the checkpoint. See the programming guide for details (constraints, etc.).
    • End-to-end semantics: This stream ensures that every records is effectively received and transformed exactly once, but gives no guarantees on whether the transformed data are outputted exactly once. For end-to-end exactly-once semantics, you have to either ensure that the output operation is idempotent, or use transactions to output records atomically. See the programming guide for more details.

    ssc

    StreamingContext object

    kafkaParams

    Kafka configuration parameters. Requires "metadata.broker.list" or "bootstrap.servers" to be set with Kafka broker(s) (NOT zookeeper servers) specified in host1:port1,host2:port2 form.

    fromOffsets

    Per-topic/partition Kafka offsets defining the (inclusive) starting point of the stream

    messageHandler

    Function for translating each message and metadata into the desired type

    Annotations

    @Experimental()

12. def createRDD[K, V, KD <: Decoder[K], VD <: Decoder[V], R](jsc: JavaSparkContext, keyClass: Class[K], valueClass: Class[V], keyDecoderClass: Class[KD], valueDecoderClass: Class[VD], recordClass: Class[R], kafkaParams: Map[String, String], offsetRanges: Array[OffsetRange], leaders: Map[TopicAndPartition, Broker], messageHandler: Function[MessageAndMetadata[K, V], R]): JavaRDD[R]

    :: Experimental :: Create a RDD from Kafka using offset ranges for each topic and partition.

    :: Experimental :: Create a RDD from Kafka using offset ranges for each topic and partition. This allows you specify the Kafka leader to connect to (to optimize fetching) and access the message as well as the metadata.

    jsc

    JavaSparkContext object

    kafkaParams

    Kafka configuration parameters. Requires "metadata.broker.list" or "bootstrap.servers" to be set with Kafka broker(s) (NOT zookeeper servers) specified in host1:port1,host2:port2 form.

    offsetRanges

    Each OffsetRange in the batch corresponds to a range of offsets for a given Kafka topic/partition

    leaders

    Kafka brokers for each TopicAndPartition in offsetRanges. May be an empty map, in which case leaders will be looked up on the driver.

    messageHandler

    Function for translating each message and metadata into the desired type

    Annotations

    @Experimental()

13. def createRDD[K, V, KD <: Decoder[K], VD <: Decoder[V]](jsc: JavaSparkContext, keyClass: Class[K], valueClass: Class[V], keyDecoderClass: Class[KD], valueDecoderClass: Class[VD], kafkaParams: Map[String, String], offsetRanges: Array[OffsetRange]): JavaPairRDD[K, V]

    Create a RDD from Kafka using offset ranges for each topic and partition.

    Create a RDD from Kafka using offset ranges for each topic and partition.

    jsc

    JavaSparkContext object

    kafkaParams

    Kafka configuration parameters. Requires "metadata.broker.list" or "bootstrap.servers" to be set with Kafka broker(s) (NOT zookeeper servers) specified in host1:port1,host2:port2 form.

    offsetRanges

    Each OffsetRange in the batch corresponds to a range of offsets for a given Kafka topic/partition

    Annotations

    @Experimental()

14. def createRDD[K, V, KD <: Decoder[K], VD <: Decoder[V], R](sc: SparkContext, kafkaParams: Map[String, String], offsetRanges: Array[OffsetRange], leaders: Map[TopicAndPartition, Broker], messageHandler: (MessageAndMetadata[K, V]) ⇒ R)(implicit arg0: ClassTag[K], arg1: ClassTag[V], arg2: ClassTag[KD], arg3: ClassTag[VD], arg4: ClassTag[R]): RDD[R]

    :: Experimental :: Create a RDD from Kafka using offset ranges for each topic and partition.

    :: Experimental :: Create a RDD from Kafka using offset ranges for each topic and partition. This allows you specify the Kafka leader to connect to (to optimize fetching) and access the message as well as the metadata.

    sc

    SparkContext object

    kafkaParams

    Kafka configuration parameters. Requires "metadata.broker.list" or "bootstrap.servers" to be set with Kafka broker(s) (NOT zookeeper servers) specified in host1:port1,host2:port2 form.

    offsetRanges

    Each OffsetRange in the batch corresponds to a range of offsets for a given Kafka topic/partition

    leaders

    Kafka brokers for each TopicAndPartition in offsetRanges. May be an empty map, in which case leaders will be looked up on the driver.

    messageHandler

    Function for translating each message and metadata into the desired type

    Annotations

    @Experimental()

15. def createRDD[K, V, KD <: Decoder[K], VD <: Decoder[V]](sc: SparkContext, kafkaParams: Map[String, String], offsetRanges: Array[OffsetRange])(implicit arg0: ClassTag[K], arg1: ClassTag[V], arg2: ClassTag[KD], arg3: ClassTag[VD]): RDD[(K, V)]

    Create a RDD from Kafka using offset ranges for each topic and partition.

    Create a RDD from Kafka using offset ranges for each topic and partition.

    sc

    SparkContext object

    kafkaParams

    Kafka configuration parameters. Requires "metadata.broker.list" or "bootstrap.servers" to be set with Kafka broker(s) (NOT zookeeper servers) specified in host1:port1,host2:port2 form.

    offsetRanges

    Each OffsetRange in the batch corresponds to a range of offsets for a given Kafka topic/partition

    Annotations

    @Experimental()

16. def createStream[K, V, U <: Decoder[_], T <: Decoder[_]](jssc: JavaStreamingContext, keyTypeClass: Class[K], valueTypeClass: Class[V], keyDecoderClass: Class[U], valueDecoderClass: Class[T], kafkaParams: Map[String, String], topics: Map[String, Integer], storageLevel: StorageLevel): JavaPairReceiverInputDStream[K, V]

    Create an input stream that pulls messages from Kafka Brokers.

    Create an input stream that pulls messages from Kafka Brokers.

    jssc

    JavaStreamingContext object

    keyTypeClass

    Key type of DStream

    valueTypeClass

    value type of Dstream

    keyDecoderClass

    Type of kafka key decoder

    valueDecoderClass

    Type of kafka value decoder

    kafkaParams

    Map of kafka configuration parameters, see http://kafka.apache.org/08/configuration.html

    topics

    Map of (topic_name -> numPartitions) to consume. Each partition is consumed in its own thread

    storageLevel

    RDD storage level.

17. def createStream(jssc: JavaStreamingContext, zkQuorum: String, groupId: String, topics: Map[String, Integer], storageLevel: StorageLevel): JavaPairReceiverInputDStream[String, String]

    Create an input stream that pulls messages from Kafka Brokers.

    Create an input stream that pulls messages from Kafka Brokers.

    jssc

    JavaStreamingContext object

    zkQuorum

    Zookeeper quorum (hostname:port,hostname:port,..).

    groupId

    The group id for this consumer.

    topics

    Map of (topic_name -> numPartitions) to consume. Each partition is consumed in its own thread.

    storageLevel

    RDD storage level.

18. def createStream(jssc: JavaStreamingContext, zkQuorum: String, groupId: String, topics: Map[String, Integer]): JavaPairReceiverInputDStream[String, String]

    Create an input stream that pulls messages from Kafka Brokers.

    Create an input stream that pulls messages from Kafka Brokers. Storage level of the data will be the default StorageLevel.MEMORY_AND_DISK_SER_2.

    jssc

    JavaStreamingContext object

    zkQuorum

    Zookeeper quorum (hostname:port,hostname:port,..)

    groupId

    The group id for this consumer

    topics

    Map of (topic_name -> numPartitions) to consume. Each partition is consumed in its own thread

19. def createStream[K, V, U <: Decoder[_], T <: Decoder[_]](ssc: StreamingContext, kafkaParams: Map[String, String], topics: Map[String, Int], storageLevel: StorageLevel)(implicit arg0: ClassTag[K], arg1: ClassTag[V], arg2: ClassTag[U], arg3: ClassTag[T]): ReceiverInputDStream[(K, V)]

    Create an input stream that pulls messages from Kafka Brokers.

    Create an input stream that pulls messages from Kafka Brokers.

    ssc

    StreamingContext object

    kafkaParams

    Map of kafka configuration parameters, see http://kafka.apache.org/08/configuration.html

    topics

    Map of (topic_name -> numPartitions) to consume. Each partition is consumed in its own thread.

    storageLevel

    Storage level to use for storing the received objects

20. def createStream(ssc: StreamingContext, zkQuorum: String, groupId: String, topics: Map[String, Int], storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK_SER_2): ReceiverInputDStream[(String, String)]

    Create an input stream that pulls messages from Kafka Brokers.

    Create an input stream that pulls messages from Kafka Brokers.

    ssc

    StreamingContext object

    zkQuorum

    Zookeeper quorum (hostname:port,hostname:port,..)

    groupId

    The group id for this consumer

    topics

    Map of (topic_name -> numPartitions) to consume. Each partition is consumed in its own thread

    storageLevel

    Storage level to use for storing the received objects (default: StorageLevel.MEMORY_AND_DISK_SER_2)

21. final def eq(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

22. def equals(arg0: Any): Boolean

    Definition Classes

    AnyRef → Any

23. def finalize(): Unit

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

24. final def getClass(): Class[_]

    Definition Classes

    AnyRef → Any

25. def hashCode(): Int

    Definition Classes

    AnyRef → Any

26. final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

27. final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

28. final def notify(): Unit

    Definition Classes

    AnyRef

29. final def notifyAll(): Unit

    Definition Classes

    AnyRef

30. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes

    AnyRef

31. def toString(): String

    Definition Classes

    AnyRef → Any

32. final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

33. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

34. final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )