Inability to create full location block

[ryan-dyer-sp]

Describe the bug
Our app needs to be able to proxy_pass certain locations to external entities. We have an existing nginx config of

location /path {
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-Forwarded-Port $server_port;
      proxy_set_header Host $host;
      proxy_read_timeout 600;
      fastcgi_read_timeout 600;
      proxy_pass https://external.url/path/;
}

I've tried creating this as a snippet filter of the http.server.location variety and including it in an httproute like:

      rules:
        - backendRefs:
            - group: ''
              kind: Service
              name: app
              port: 8080
              weight: 1
          matches:
            - path:
                value: /
                type: PathPrefix
        - filters:
            - extensionRef:
                group: gateway.nginx.org
                kind: SnippetsFilter
                name: external-proxy-pass
              type: ExtensionRef
          matches:
            - path:
                type: PathPrefix
                value: /path/

But this results in an invalid nginx configuration due to multiple proxy_pass directives. I'm unable to obtain the config as it is getting rolled back to the last working one.

An additional oddity is that even though the config has been rolled back, ALL HTTPRoutes using this gateway are broken and external-dns in turn wipes all the DNS entries.

To Reproduce
Steps to reproduce the behavior:
see above

Expected behavior
able to create a custom location block

Your environment

• Version of the NGINX Gateway Fabric - release version or a specific commit. The first line of the nginx-gateway container logs includes the commit info.
  NGF 2.1.0

• Version of Kubernetes
  1.32.6

• Kubernetes platform (e.g. Mini-kube or GCP)
  EKS

[nginx-bot]

Hi @ryan-dyer-sp! Welcome to the project! 🎉

Thanks for opening this issue!
Be sure to check out our Contributing Guidelines and the Issue Lifecycle while you wait for someone on the team to take a look at this.

[bjee19]

Hey @ryan-dyer-sp, thanks for reporting this issue. Regarding this:

Our app needs to be able to proxy_pass certain locations to external entities.

We recently completed this issue on Supporting ExternalName Services in this PR. Does that cover your use case?

Also, we will be investigating this:

An additional oddity is that even though the config has been rolled back, ALL HTTPRoutes using this gateway are broken and external-dns in turn wipes all the DNS entries.

[cmbankester]

Also, we will be investigating this:

An additional oddity is that even though the config has been rolled back, ALL HTTPRoutes using this gateway are broken and external-dns in turn wipes all the DNS entries.

I have seen the http-routes-broken thing as well, while upgrading from 2.0.X to 2.1.X. We have several clusters running NGF, and it happened in about half of them (after the first time it happened, I started scaling-down external-dns before the upgrade to prevent DNS from getting wiped out). What I saw was this: the newly-updated control-plane pod running 2.1.X would attempt to update the configuration for a data-plan pod running 2.0.X while it was terminating, and the attempt would fail with this error:

msg: Config apply failed, rolling back config; error: error creating directory /etc/nginx/events-includes: mkdir /etc/nginx/events-includes: read-only file system

and stacktrace:

github.com/nginx/nginx-gateway-fabric/v2/internal/controller.(*eventHandlerImpl).waitForStatusUpdates
    /home/runner/work/nginx-gateway-fabric/nginx-gateway-fabric/internal/controller/handler.go:262

After that occurred, the HTTPRoutes in the cluster would all get Accepted: false with a reason mentioning something about nginx configuration not being valid (sorry I didn't copy the text last time I saw it)

[sjberman]

@cmbankester Not a great upgrade experience, sorry about that. I was aware of that error during an upgrade, because of some incompatible changes between 2.0 and 2.1 in the configuration, but didn't realize that it would result in the routes being marked as invalid during that process.

Were you able to successfully upgrade to 2.1 in the end? I'm also curious, does external-dns simply key in on the HTTPRoute condition Accepted: false? Is there anything else it cares about?

[sjberman]

I've created a couple of tickets #3866, #3867 to address these problems.

[cmbankester]

@cmbankester Not a great upgrade experience, sorry about that. I was aware of that error during an upgrade, because of some incompatible changes between 2.0 and 2.1 in the configuration, but didn't realize that it would result in the routes being marked as invalid during that process.

Were you able to successfully upgrade to 2.1 in the end? I'm also curious, does external-dns simply key in on the HTTPRoute condition Accepted: false? Is there anything else it cares about?

Yep, after I noticed what happened with the first cluster, I began stopping external-dns before doing the upgrade on followup clusters and I was able to get them all updated without incident. Thanks for the followup!