Add documentation for Kafka Proxy #33911
Conversation
|
Thank you for your submission! We really appreciate it. Like many source-available projects, we require that you sign our Contributor License Agreement (CLA) before we can accept your contribution. I have read the Contributor License Agreement (CLA) and I hereby sign the CLA. You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot. |
| [kafka-proxy](https://github.com/grepplabs/kafka-proxy) by default is configured | ||
| to handle multiple brokers in the upstream kafka topic by dynamically creating | ||
| listeners on new ports on the proxy instance. It is then expected for kafka | ||
| clients to start reading off of these ports. Utilizing an | ||
| [SSH Bastion](/ingest-data/network-security/ssh-tunnel/) will allow | ||
| Materialize to properly respond to the dynamic ports as they are created. |
There was a problem hiding this comment.
Does this mean you can only use dynamic listeners with an ssh bastion? I.e, you can't connect directly if the proxy is what exposes your cluster to the public intenet?
There was a problem hiding this comment.
So you can connect if it's exposed to the public internet, I've added a section for this. In theory you could do privatelink if you had a script or program that dynamically registered listeners to target groups on an NLB. I've not tested this though, happy to document it or just leave it off for now.
| {{< tabs tabID="1" >}} | ||
| {{< tab "kafka-proxy" >}} | ||
|
|
There was a problem hiding this comment.
I had initially planned on using them for covering different patterns, i.e. reverse-proxy as well, but will remove them for now.
| ### Working with [kafka-proxy](https://github.com/grepplabs/kafka-proxy) | ||
|
|
||
| When working with [kafka-proxy](https://github.com/grepplabs/kafka-proxy) | ||
| there are a few patterns to consider when it comes to listener configuration | ||
| and network security. |
There was a problem hiding this comment.
The general flow of this section is a little confusing to me. Each should start with what the thing is - dynamic listener or static listener - and then how to use it with Materialize. The lead in with ssh feels abrupt.
Adding documentation for how to connect to kafka-proxy as a source.