|
| 1 | +--- |
| 2 | +title: Kafka Destination |
| 3 | +--- |
| 4 | + |
| 5 | +{% include content/plan-grid.md name="actions" %} |
| 6 | + |
| 7 | +[Kafka](https://kafka.apache.org/?utm_source=segmentio&utm_medium=docs&utm_campaign=partners){:target="_blank”} provides a highly scalable and fault-tolerant messaging system that enables real-time data processing and stream processing at scale. When integrated with Segment, Kafka serves as a powerful backbone for managing and processing event data collected by Segment, allowing businesses to efficiently ingest, route, and analyze data across various applications and systems in real time. |
| 8 | + |
| 9 | +This destination is maintained by Segment. |
| 10 | + |
| 11 | +## Getting started |
| 12 | + |
| 13 | +### Create the Kafka Destination |
| 14 | + |
| 15 | +1. From your workspace's [Destination catalog page](https://app.segment.com/goto-my-workspace/destinations/catalog){:target="_blank”} search for "Kafka" |
| 16 | +2. Select the "Kafka" tile and click **Add Destination** |
| 17 | +3. Select an existing Source to connect to Kafka. |
| 18 | +4. Give your Kafka Destination a name. |
| 19 | + |
| 20 | +### Configure the Kafka Destination |
| 21 | + |
| 22 | +Authentication and Encryption settings for Kafka depend on the way your cluster is configured. You may need the assistance of someone technical to provide values for the following configuration Settings: |
| 23 | + |
| 24 | +5. On the Settings tab, provide values for the **Client ID**, **Brokers** and **Authentication Mechanism** Setting fields. |
| 25 | +6. Populate fields for your selected **Authentication Mechanism**: |
| 26 | + For **Plain** or **SCRAM-SHA-256 / 512**: provide values for **Username** and **Password** fields. |
| 27 | + For **AWS**: Provide values for **AWS Access Key ID** and **AWS Secret Key** fields, and optionally for the **AWS Authorization Identity** field. |
| 28 | + For **Client Certificate** you will need to provide values for the **SSL Client Key** and **SSL Client Certificate** Settings fields. |
| 29 | +7. Optionally populate the **SSL Certificate Authority** field. |
| 30 | +8. Save your changes before proceeding to the next step. |
| 31 | + |
| 32 | +### Configure the "Send" Action |
| 33 | + |
| 34 | +9. Click to the Mappings tab and add a new **Send** Mapping. |
| 35 | +10. Select a Topic to send data to. This field should auto-populate based on the credentials provided in the Settings tab. |
| 36 | +11. Map your payload using the **Payload** field. |
| 37 | +12. Optionally specify Partitioning preferences, Headers and Message Key values. |
| 38 | +13. Save and enable the Action, then navigate back to the Destination's Settings tab and enable and save the Destination. |
| 39 | + |
| 40 | +The Destination should now be configured to send data to your Kafka cluster. |
| 41 | + |
| 42 | +{% include components/actions-fields.html %} |
| 43 | + |
| 44 | +## FAQ |
| 45 | + |
| 46 | +### Which Kafka Platforms are supported? |
| 47 | + |
| 48 | +The Kafka Destination can send data to Topics on self hosted Kafka Clusters, or to Clusters hosted on Managed Service platforms such as **Confluent Cloud** and **Aiven**. |
| 49 | + |
| 50 | +### Which data formats are supported? |
| 51 | + |
| 52 | +Data will be sent to Kafka in JSON format only. Avro, Protobuf and other formats are not yet supported. |
| 53 | + |
| 54 | +### Which authentication formats are supported? |
| 55 | + |
| 56 | +The following SASL based authentication mechanisms are supported: **Plain**, **SCRAM-SHA-256**, **SCRAM-SHA-512** and **AWS**. |
| 57 | +**Client Certificate** auth is also supported. |
| 58 | + |
| 59 | +### How is Partitioning controlled? |
| 60 | + |
| 61 | +The **Send** Action provides allows for multiple ways to specify which an event should be sent to. |
| 62 | + |
| 63 | +1) The **Partition** field: Use this field to specify the name of the Partition to send the event to. |
| 64 | +2) The **Default Partition** field: Use this field to specify a default Partition. This will be used when value is provided in the **Partition** field. |
| 65 | +3) The **Message Key** field: A hash of the field's value will be used to determine which Partition to send the event to. If no value is provided in the **Message Key** field a round robin algorithm is used. |
| 66 | + |
| 67 | +### What is the "SSL - Reject Unauthorized Certificate Authority" field for? |
| 68 | + |
| 69 | +This field specifies if Segment should reject server connections when a certificate is notsigned by a trusted Certificate Authority (CA). This can be useful for testing purposes or when using a self-signed certificate. |
0 commit comments