|
1 | 1 | ---
|
2 | 2 | title: Kafka Destination
|
| 3 | +beta: true |
| 4 | +hidden: true |
| 5 | +id: |
3 | 6 | ---
|
4 | 7 |
|
5 | 8 | {% include content/plan-grid.md name="actions" %}
|
6 | 9 |
|
7 | 10 | [Kafka](https://kafka.apache.org/?utm_source=segmentio&utm_medium=docs&utm_campaign=partners){:target="_blank”} provides a highly scalable and fault-tolerant messaging system that enables real-time data processing and stream processing at scale. When integrated with Segment, Kafka serves as a powerful backbone for managing and processing event data collected by Segment, allowing businesses to efficiently ingest, route, and analyze data across various applications and systems in real time.
|
8 | 11 |
|
9 |
| -This destination is maintained by Segment. |
| 12 | +This destination is maintained by Segment. For any issues with the destination, [contact the Segment Support team](mailto:[email protected]). |
10 | 13 |
|
11 | 14 | ## Getting started
|
12 | 15 |
|
13 | 16 | ### Create the Kafka Destination
|
14 | 17 |
|
15 |
| -1. From your workspace's [Destination catalog page](https://app.segment.com/goto-my-workspace/destinations/catalog){:target="_blank”} search for "Kafka" |
16 |
| -2. Select the "Kafka" tile and click **Add Destination** |
| 18 | +1. From your workspace's [Destination catalog page](https://app.segment.com/goto-my-workspace/destinations/catalog){:target="_blank”} search for "Kafka". |
| 19 | +2. Select the "Kafka" tile and click **Add Destination**. |
17 | 20 | 3. Select an existing Source to connect to Kafka.
|
18 |
| -4. Give your Kafka Destination a name. |
| 21 | +4. Enter a name for your Kafka destination. |
19 | 22 |
|
20 | 23 | ### Configure the Kafka Destination
|
21 | 24 |
|
22 |
| -Authentication and Encryption settings for Kafka depend on the way your cluster is configured. You may need the assistance of someone technical to provide values for the following configuration Settings: |
| 25 | +The way you've configured your Kafka Cluster informs the authentication and encryption settings you'll need to apply to the Segment Kafka Destination. You may need the assistance of someone technical to provide values for the following Settings: |
23 | 26 |
|
24 |
| -5. On the Settings tab, provide values for the **Client ID**, **Brokers** and **Authentication Mechanism** Setting fields. |
25 |
| -6. Populate fields for your selected **Authentication Mechanism**: |
26 |
| - For **Plain** or **SCRAM-SHA-256 / 512**: provide values for **Username** and **Password** fields. |
27 |
| - For **AWS**: Provide values for **AWS Access Key ID** and **AWS Secret Key** fields, and optionally for the **AWS Authorization Identity** field. |
28 |
| - For **Client Certificate** you will need to provide values for the **SSL Client Key** and **SSL Client Certificate** Settings fields. |
29 |
| -7. Optionally populate the **SSL Certificate Authority** field. |
30 |
| -8. Save your changes before proceeding to the next step. |
| 27 | +5. On the Settings tab, enter values into the **Client ID**, **Brokers** and **Authentication Mechanism** setting fields. |
| 28 | +6. Populate fields based on the value you selected from the **Authentication Mechanism** field: |
| 29 | +- **Plain** or **SCRAM-SHA-256 / 512** authentication: provide values for **Username** and **Password** fields. |
| 30 | +- **AWS** authentication: provide values for **AWS Access Key ID** and **AWS Secret Key** fields, and optionally for the **AWS Authorization Identity** field. |
| 31 | +- **Client Certificate** authentication: provide values for the **SSL Client Key** and **SSL Client Certificate** fields. |
| 32 | +7. Populate the **SSL Certificate Authority** field, if necessary. |
| 33 | +8. Save your changes and proceed to Configure the Send Action. |
31 | 34 |
|
32 | 35 | ### Configure the "Send" Action
|
33 | 36 |
|
34 |
| -9. Click to the Mappings tab and add a new **Send** Mapping. |
35 |
| -10. Select a Topic to send data to. This field should auto-populate based on the credentials provided in the Settings tab. |
| 37 | +9. Select the Mappings tab and add a new **Send** mapping. |
| 38 | +10. Select a Topic to send data to. This field should auto-populate based on the credentials you provided in the Settings tab. |
36 | 39 | 11. Map your payload using the **Payload** field.
|
37 |
| -12. Optionally specify Partitioning preferences, Headers and Message Key values. |
38 |
| -13. Save and enable the Action, then navigate back to the Destination's Settings tab and enable and save the Destination. |
39 |
| - |
40 |
| -The Destination should now be configured to send data to your Kafka cluster. |
| 40 | +12. _(Optional)_: Specify partitioning preferences, Headers and Message Key values. |
| 41 | +13. Save and enable the Action, then navigate back to the Kafka destination's Settings tab to enable and save the Destination. |
41 | 42 |
|
42 | 43 | {% include components/actions-fields.html %}
|
43 | 44 |
|
44 | 45 | ## FAQ
|
45 | 46 |
|
46 | 47 | ### Which Kafka Platforms are supported?
|
47 | 48 |
|
48 |
| -The Kafka Destination can send data to Topics on self hosted Kafka Clusters, or to Clusters hosted on Managed Service platforms such as **Confluent Cloud** and **Aiven**. |
| 49 | +The Kafka Destination can send data to Topics on self-hosted Kafka Clusters, or to Clusters hosted on Managed Service platforms like **Confluent Cloud** and **Aiven** |
49 | 50 |
|
50 | 51 | ### Which data formats are supported?
|
51 | 52 |
|
52 |
| -Data will be sent to Kafka in JSON format only. Avro, Protobuf and other formats are not yet supported. |
| 53 | +Segment sends data to Kafka in JSON format only. Segment does not yet support other formats, like Avro or Protobuf. |
53 | 54 |
|
54 | 55 | ### Which authentication mechanisms are supported?
|
55 | 56 |
|
56 | 57 | The Authentication Mechanism is controlled with the **Authentication Mechanism** Setting field.
|
57 | 58 |
|
58 |
| -The following SASL based authentication mechanisms are supported: **Plain**, **SCRAM-SHA-256**, **SCRAM-SHA-512** and **AWS**. |
59 |
| -**Client Certificate** auth is also supported. |
| 59 | +Segment supports the following SASL-based authentication methods: |
| 60 | +- Plain |
| 61 | +- SCRAM-SHA-256 |
| 62 | +- SCRAM-SHA-512 |
| 63 | +- AWS |
| 64 | + |
| 65 | +Segment also supports **Client Certificate** authentication. |
60 | 66 |
|
61 |
| -### How is Partitioning controlled? |
| 67 | +### How is partitioning controlled? |
62 | 68 |
|
63 | 69 | The **Send** Action provides multiple ways to specify which Partition an event should be sent to.
|
64 | 70 |
|
65 |
| -1) The **Partition** field: Use this field to specify the name of the Partition to send the event to. |
66 |
| -2) The **Default Partition** field: Use this field to specify a default Partition. This will be used when value is provided in the **Partition** field. |
67 |
| -3) The **Message Key** field: A hash of the field's value will be used to determine which Partition to send the event to. If no value is provided in the **Message Key** field a round robin algorithm is used. |
| 71 | +- **Partition**: Use this field to specify the name of the Partition Segment should send events to. |
| 72 | +- **Default Partition**: Use this field to specify a default Partition. Segment uses this when you don't provide a value in the **Partition** field. |
| 73 | +- **Message Key**: Segment uses a hash of this field's value to determine which Partition should receive an event. If you don't provide a Message Key, Segment uses a round robin algorithm to select the partition to send the event to. |
68 | 74 |
|
69 | 75 | ### What is the "SSL - Reject Unauthorized Certificate Authority" field for?
|
70 | 76 |
|
|
0 commit comments