Skip to content

Commit 371fc8f

Browse files
adding Kafka Destination docs
1 parent 00772d4 commit 371fc8f

File tree

1 file changed

+69
-0
lines changed
  • src/connections/destinations/catalog/kafka

1 file changed

+69
-0
lines changed
Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
---
2+
title: Kafka Destination
3+
---
4+
5+
{% include content/plan-grid.md name="actions" %}
6+
7+
[Kafka](https://kafka.apache.org/?utm_source=segmentio&utm_medium=docs&utm_campaign=partners){:target="_blank”} provides a highly scalable and fault-tolerant messaging system that enables real-time data processing and stream processing at scale. When integrated with Segment, Kafka serves as a powerful backbone for managing and processing event data collected by Segment, allowing businesses to efficiently ingest, route, and analyze data across various applications and systems in real time.
8+
9+
This destination is maintained by Segment.
10+
11+
## Getting started
12+
13+
### Create the Kafka Destination
14+
15+
1. From your workspace's [Destination catalog page](https://app.segment.com/goto-my-workspace/destinations/catalog){:target="_blank”} search for "Kafka"
16+
2. Select the "Kafka" tile and click **Add Destination**
17+
3. Select an existing Source to connect to Kafka.
18+
4. Give your Kafka Destination a name.
19+
20+
### Configure the Kafka Destination
21+
22+
Authentication and Encryption settings for Kafka depend on the way your cluster is configured. You may need the assistance of someone technical to provide values for the following configuration Settings:
23+
24+
5. On the Settings tab, provide values for the **Client ID**, **Brokers** and **Authentication Mechanism** Setting fields.
25+
6. Populate fields for your selected **Authentication Mechanism**:
26+
For **Plain** or **SCRAM-SHA-256 / 512**: provide values for **Username** and **Password** fields.
27+
For **AWS**: Provide values for **AWS Access Key ID** and **AWS Secret Key** fields, and optionally for the **AWS Authorization Identity** field.
28+
For **Client Certificate** you will need to provide values for the **SSL Client Key** and **SSL Client Certificate** Settings fields.
29+
7. Optionally populate the **SSL Certificate Authority** field.
30+
8. Save your changes before proceeding to the next step.
31+
32+
### Configure the "Send" Action
33+
34+
9. Click to the Mappings tab and add a new **Send** Mapping.
35+
10. Select a Topic to send data to. This field should auto-populate based on the credentials provided in the Settings tab.
36+
11. Map your payload using the **Payload** field.
37+
12. Optionally specify Partitioning preferences, Headers and Message Key values.
38+
13. Save and enable the Action, then navigate back to the Destination's Settings tab and enable and save the Destination.
39+
40+
The Destination should now be configured to send data to your Kafka cluster.
41+
42+
{% include components/actions-fields.html %}
43+
44+
## FAQ
45+
46+
### Which Kafka Platforms are supported?
47+
48+
The Kafka Destination can send data to Topics on self hosted Kafka Clusters, or to Clusters hosted on Managed Service platforms such as **Confluent Cloud** and **Aiven**.
49+
50+
### Which data formats are supported?
51+
52+
Data will be sent to Kafka in JSON format only. Avro, Protobuf and other formats are not yet supported.
53+
54+
### Which authentication formats are supported?
55+
56+
The following SASL based authentication mechanisms are supported: **Plain**, **SCRAM-SHA-256**, **SCRAM-SHA-512** and **AWS**.
57+
**Client Certificate** auth is also supported.
58+
59+
### How is Partitioning controlled?
60+
61+
The **Send** Action provides allows for multiple ways to specify which an event should be sent to.
62+
63+
1) The **Partition** field: Use this field to specify the name of the Partition to send the event to.
64+
2) The **Default Partition** field: Use this field to specify a default Partition. This will be used when value is provided in the **Partition** field.
65+
3) The **Message Key** field: A hash of the field's value will be used to determine which Partition to send the event to. If no value is provided in the **Message Key** field a round robin algorithm is used.
66+
67+
### What is the "SSL - Reject Unauthorized Certificate Authority" field for?
68+
69+
This field specifies if Segment should reject server connections when a certificate is notsigned by a trusted Certificate Authority (CA). This can be useful for testing purposes or when using a self-signed certificate.

0 commit comments

Comments
 (0)