|
| 1 | +--- |
| 2 | +title: Kafka Destination |
| 3 | +beta: true |
| 4 | +hidden: true |
| 5 | +id: 65dde5755698cb0dab09b489 |
| 6 | +--- |
| 7 | + |
| 8 | +{% include content/plan-grid.md name="actions" %} |
| 9 | + |
| 10 | +[Kafka](https://kafka.apache.org/?utm_source=segmentio&utm_medium=docs&utm_campaign=partners){:target="_blank”} provides a highly scalable and fault-tolerant messaging system that enables real-time data processing and stream processing at scale. When integrated with Segment, Kafka serves as a powerful backbone for managing and processing event data collected by Segment, allowing businesses to efficiently ingest, route, and analyze data across various applications and systems in real time. |
| 11 | + |
| 12 | +This destination is maintained by Segment. For any issues with the destination, [contact the Segment Support team ](mailto:[email protected]). |
| 13 | + |
| 14 | +## Getting started |
| 15 | + |
| 16 | +### Create the Kafka Destination |
| 17 | + |
| 18 | +1. From your workspace's [Destination catalog page](https://app.segment.com/goto-my-workspace/destinations/catalog){:target="_blank”} search for "Kafka". |
| 19 | +2. Select the "Kafka" tile and click **Add Destination**. |
| 20 | +3. Select an existing Source to connect to Kafka. |
| 21 | +4. Enter a name for your Kafka destination. |
| 22 | + |
| 23 | +### Configure the Kafka Destination |
| 24 | + |
| 25 | +The way you've configured your Kafka Cluster informs the authentication and encryption settings you'll need to apply to the Segment Kafka Destination. You may need the assistance of someone technical to provide values for the following Settings: |
| 26 | + |
| 27 | +<ol style="counter-reset: none"> |
| 28 | + <li value="5" markdown="1"> |
| 29 | + On the Settings tab, enter values into the **Client ID**, **Brokers** and **Authentication Mechanism** setting fields. |
| 30 | + </li> |
| 31 | + <li value="6"> |
| 32 | + Populate fields based on the value you selected from the <b>Authentication Mechanism</b> field: |
| 33 | + <ul> |
| 34 | + <li> |
| 35 | + <b>Plain</b> or <b>SCRAM-SHA-256 / 512</b> authentication: provide values for <b>Username</b> and <b>Password</b> fields. |
| 36 | + </li> |
| 37 | + <li> |
| 38 | + <b>AWS</b> authentication: provide values for <b>AWS Access Key ID</b> and <b>AWS Secret Key</b> fields, and optionally for the <b>AWS Authorization Identity</b> field. |
| 39 | + </li> |
| 40 | + <li> |
| 41 | + <b>Client Certificate</b> authentication: provide values for the <b>SSL Client Key</b> and <b>SSL Client Certificate</b> fields. |
| 42 | + </li> |
| 43 | + </ul> |
| 44 | + </li> |
| 45 | + <li value="7" markdown="1"> |
| 46 | + Populate the **SSL Certificate Authority** field, if necessary. |
| 47 | + </li> |
| 48 | + <li value="8" markdown="1"> |
| 49 | + Save your changes and proceed to [Configure the Send Action](#configure-the-send-action). |
| 50 | + </li> |
| 51 | +</ol> |
| 52 | + |
| 53 | +### Configure the "Send" Action |
| 54 | + |
| 55 | +<ol style="counter-reset: none;"> |
| 56 | + <li value="9" markdown=1> |
| 57 | + Select the Mappings tab and add a new **Send** mapping. |
| 58 | + </li> |
| 59 | + <li value="10" markdown=1> |
| 60 | + Select a Topic to send data to. This field should auto-populate based on the credentials you provided in the Settings tab. |
| 61 | + </li> |
| 62 | + <li value="11" markdown=1> |
| 63 | + Map your payload using the **Payload** field. <br> _(Optional)_: Specify partitioning preferences, Headers and Message Key values. |
| 64 | + </li> |
| 65 | + <li value="12" markdown=1> |
| 66 | + Save and enable the Action, then navigate back to the Kafka destination's Settings tab to enable and save the Destination. |
| 67 | + </li> |
| 68 | +</ol> |
| 69 | + |
| 70 | +{% include components/actions-fields.html %} |
| 71 | + |
| 72 | +## FAQ |
| 73 | + |
| 74 | +### Which Kafka Platforms are supported? |
| 75 | + |
| 76 | +The Kafka Destination can send data to Topics on self-hosted Kafka Clusters, or to Clusters hosted on Managed Service platforms like **Confluent Cloud** and **Aiven**. |
| 77 | + |
| 78 | +### Which data formats are supported? |
| 79 | + |
| 80 | +Segment sends data to Kafka in JSON format only. Segment does not yet support other formats, like Avro or Protobuf. |
| 81 | + |
| 82 | +### Which authentication mechanisms are supported? |
| 83 | + |
| 84 | +The Authentication Mechanism is controlled with the **Authentication Mechanism** Setting field. |
| 85 | + |
| 86 | +Segment supports the following SASL-based authentication methods: |
| 87 | +- Plain |
| 88 | +- SCRAM-SHA-256 |
| 89 | +- SCRAM-SHA-512 |
| 90 | +- AWS |
| 91 | + |
| 92 | +Segment also supports **Client Certificate** authentication. |
| 93 | + |
| 94 | +### How is partitioning controlled? |
| 95 | + |
| 96 | +The **Send** Action provides multiple ways to specify which Partition an event should be sent to. |
| 97 | + |
| 98 | +- **Partition**: Use this field to specify the name of the Partition Segment should send events to. |
| 99 | +- **Default Partition**: Use this field to specify a default Partition. Segment uses this when you don't provide a value in the **Partition** field. |
| 100 | +- **Message Key**: Segment uses a hash of this field's value to determine which Partition should receive an event. If you don't provide a Message Key, Segment uses a round robin algorithm to select the partition to send the event to. |
| 101 | + |
| 102 | +### What is the "SSL - Reject Unauthorized Certificate Authority" field for? |
| 103 | + |
| 104 | +This field specifies if Segment should reject server connections when a certificate is notsigned by a trusted Certificate Authority (CA). This can be useful for testing purposes or when using a self-signed certificate. |
0 commit comments