ruby-kafka

Kafka::Client#fetch_messages

Fetches a batch of messages from a single partition. Note that it's possible
to get back empty batches.

The starting point for the fetch can be configured with the `:offset` argument.
If you pass a number, the fetch will start at that offset. However, there are
two special Symbol values that can be passed instead:

* `:earliest` — the first offset in the partition.
* `:latest` — the next offset that will be written to, effectively making the
  call block until there is a new message in the partition.

The Kafka protocol specifies the numeric values of these two options: -2 and -1,
respectively. You can also pass in these numbers directly.

## Example

When enumerating the messages in a partition, you typically fetch batches
sequentially.

    offset = :earliest

    loop do
      messages = kafka.fetch_messages(
        topic: "my-topic",
        partition: 42,
        offset: offset,
      )

      messages.each do |message|
        puts message.offset, message.key, message.value

        # Set the next offset that should be read to be the subsequent
        # offset.
        offset = message.offset + 1
      end
    end

See a working example in `examples/simple-consumer.rb`.

Source | Google | Stack overflow