Unable to set proxy_protocol for TransportServer

[wouterh-dev]

Version

edge

What Kubernetes platforms are you running on?

Openshift

Steps to reproduce

makeTransportListener hardcodes proxyProtocol to false so it seems impossible to let nginx use proxy_protocol for TransportServers

[github-actions]

Hi @wouterh-dev thanks for reporting!

Be sure to check out the docs and the Contributing Guidelines while you wait for a human to take a look at this 🙂

Cheers!

[pdabelf5]

@wouterh-dev Thank you for highlighting this. As described in our example, proxy-protocol should be available for TransportServer with TLS Passthrough. We will look to address this issue in a future release.

[wouterh-dev]

Thanks for pointing that out, I didn't want to enable SSL Passthrough since it's not relevant to my situation. I wound up overriding the template:

22,26c22,27
<         {{- if $s.TLSPassthrough }}
<     listen {{ $s.UnixSocket }} proxy_protocol;
<     set_real_ip_from unix:;
<         {{- else }}
<     {{ makeTransportListener $s | printf }}
---
>     listen {{ $s.Port }} proxy_protocol;
>     listen [::]:{{ $s.Port }} proxy_protocol;
>     set_real_ip_from 10.250.0.3;
> 
28d28
<         {{- end }}

Not ideal but it works for my deployment. I think it would make sense that if proxy-protocol is set to true this is also applied to makeTransportListener, and it would make sense for the set-real-ip-from setting to be applied as well.

[pdabelf5]

If you could expand on your use case, we can look into creating an enhancement issue for TransportServer

[wouterh-dev]

My use case is using the nginx ingress on an OpenStack Kubernetes cluster with Amphora load balancers which require the use of the PROXY protocol to retrieve source addresses.

I want to use Nginx as the ingress for various https services (without SSL Passthrough), and for plain TCP services as well.

My desire to use Nginx for TCP forwarding/loadbalancing is related to kubernetes/cloud-provider-openstack only allowing two Kubernetes Services to share the same OpenStack LoadBalancer (kubernetes/cloud-provider-openstack#2978), so instead I decided to use use nginx which is already bound to the OpenStack LoadBalancer.

But there are other reasons for wanting to use nginx as a tcp load balancer as well, since this is provider specific functionality anyway.

So I am using the nginx ingress like this:

controller:
  service:
    annotations:
      loadbalancer.openstack.org/proxy-protocol: "true"
      loadbalancer.openstack.org/keep-floatingip: "true"
    customPorts:
      - port: 1234
        targetPort: 1234
        protocol: TCP
        name: my-tcp-service
  globalConfiguration:
    create: true
    spec:
      listeners:
        - name: my-tcp-service
          port: 1234
          protocol: TCP
  config:
    entries:
      "proxy-protocol": "true"
      "set-real-ip-from": "10.250.0.3"
      "real-ip-header": "proxy_protocol"

Because OpenStack's Amphora LoadBalancer requires the use of the PROXY protocol to retrieve source IP addresses I have to set loadbalancer.openstack.org/proxy-protocol: "true". That means that all traffic received by the LoadBalancer Service created for nginx will use the PROXY protocol.

Nginx handles this for port 80 & 443 because of "proxy-protocol": "true", and the source IP is correct because of "set-real-ip-from": "10.250.0.3" & "real-ip-header": "proxy_protocol".

This works great for HTTP & HTTPS services managed via Ingresses. But this is broken for TransportServers, because the following nginx configuration is generated:

     listen 1234;
     listen [::]:1234;

This will receive TCP traffic using the PROXY protocol, but doesn't interpret it as such. So the traffic that gets forwarded to the target service is also wrapped in the PROXY protocol, as added by Amphora and should have been interpreted by nginx.

So it would make sense to generate this instead:

     listen 1234 proxy_protocol; # proxy_protocol because `"proxy-protocol": "true"`
     listen [::]:1234 proxy_protocol; # proxy_protocol because `"proxy-protocol": "true"`
     set_real_ip_from 10.250.0.3; # because "set-real-ip-from": "10.250.0.3"

[pdabelf5]

Thank you @wouterh-dev for the detailed response. This will certainly help us when prioritising this issue.