Kafka Output

Processed messages can be sent to a Kafka cluster, by mentioning the kafka output type in the configuration file:

[output]
type = "kafka"
kafka_brokers = [ "172.16.205.129:9092", "172.16.205.130:9092" ]
kafka_topic = "test"

The Kafka brokers (kafka_broker) and topic (kafka_topic) are mandatory.

Option properties:

• kafka_threads = <number>: dedicate the chosen number of threads to the Kafka injection process. By default, a single thread is reserved for that purpose.
• kafka_acks = <0 | 1 | -1>: controls whether Flowgger waits for an acknowledgment from the Kafka broker after having sent a batch. If you want to favor speed, use kafka_acks = 0.
• kafka_timeout = <milliseconds>: give up after Kafka has been unreachable for milliseconds.
• kafka_coalesce = <number>: with kafka_coalesce set to <number>, writes to Kafka will happen in batches of <number> records. If your traffic rate is fairly high, setting this to 10000 is a reasonable ballpark figure. However, Flowgger always waits for a full batch to be buffered before sending it to Kafka. When debugging or if your incoming traffic rate is low, you should disable coalescing by setting kafka_coalesce to 1.
• kafka_compression = "none|gzip|snappy": enable Kafka compression. This reduces bandwidth requirements, but is way more taxing on the CPU.