modsecurity: Including custom rules file throws error #11531

konmedia-devops · 2024-07-02T07:44:13Z

What happened:

As the annotation nginx.ingress.kubernetes.io/modsecurity-snippet has a limited length (afaik 4000 Bytes) and our ruleset has grown and exceeds that limit, I've been trying to centralize our ruleset.

So, I put our custom rulesets in the configmap und mounted them into ingress as a volume:

Configmap

apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-controller
  namespace: kube-system
  uid: 4080f40b-efe3-4f15-83eb-827c4383b5f6
  resourceVersion: '254309810'
  creationTimestamp: '2023-09-19T12:49:46Z'
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.10.0
    helm.sh/chart: ingress-nginx-4.10.0
    k8slens-edit-resource-version: v1
  annotations:
    meta.helm.sh/release-name: ingress-nginx
    meta.helm.sh/release-namespace: kube-system
data:
  allow-snippet-annotations: 'true'
  custom-default-ruleset: >
    SecRule REQUEST_METHOD \"^HEAD$\"
    \"id:1001,phase:2,deny,status:403,msg:\'Access denied - HEAD request not
    allowed\'\"

    SecRule REQUEST_METHOD \"^PATCH$\"
    \"id:1002,phase:2,deny,status:403,msg:\'Access denied - PATCH$ request not
    allowed\'\"
  custom-extended-ruleset: >-
    SecRule REQUEST_URI \"@beginsWith /login\"
    \"id:1100,phase:2,chain,deny,msg:\'Invalid login request\'\"
      SecRule REQUEST_URI \"!@rx \/login(\~?[a-z]{0,2}\_?[A-Z]{0,2}\/?)$\" \"id:1101,phase:2,chain\"
        SecRule REQUEST_URI \"!@rx \/login(\~?[a-z]{0,2}\_?[A-Z]{0,2}\/?)\?loggedOut=true$\" \"id:1102,phase:2,chain\"
          SecRule REQUEST_URI \"!@rx \/login(\~?[a-z]{0,2}\_?[A-Z]{0,2}\/?)\?targetUrl=([a-f0-9]*)$\" \"id:1103,phase:2,chain\"
            SecRule REQUEST_URI \"!@rx \/login(\~?[a-z]{0,2}\_?[A-Z]{0,2}\/?)\?authenticationRequired=true\" \"id:1104,phase:2,chain\"
              SecRule REQUEST_URI \"!@rx \/login(\~?[a-z]{0,2}\_?[A-Z]{0,2}\/?)(([\??\&?])(request|execute)=([a-zA-Z]+)\/([a-zA-Z]+)\.([a-z\/]+)){1,2}\"

  custom-rule-removals: "SecRule REQUEST_URI \\\"@beginsWith /admin?request=Component/Pwa.none&execute=Base/Edit.json/save\\\" \\\"id:2000,phase:1,pass,nolog,ctl:ruleRemoveById=941100,ctl:ruleRemoveById=941160,ctl:ruleRemoveById=941310\\\"\nSecRule REQUEST_URI \\\"@beginsWith /admin?execute=Editor/H5e.save/w2p/template\\\" \\\"id:2001,phase:1,pass,nolog,ctl:ruleRemoveById=941100,ctl:ruleRemoveById=941160,ctl:ruleRemoveById=941310\\\"\t\t\t\nSecRule REQUEST_URI \\\"@rx \\/(admin|editorH5e~([a-z]{2})_([A-Z]{2})|editor-h5e\\/)\\?execute=Editor\\/H5e\\.create\\/preview\\/pdf\\\" \\\"id:2002,phase:1,pass,nolog,ctl:ruleRemoveById=941100,ctl:ruleRemoveById=941160,ctl:ruleRemoveById=941310\\\"\nSecRule REQUEST_URI \\\"@rx \\/(admin|editorH5e~([a-z]{2})_([A-Z]{2})|editor-h5e\\/)\\?execute=Editor\\/H5e\\.update\\\" \\\"id:2003,phase:1,pass,nolog,ctl:ruleRemoveById=941100,ctl:ruleRemoveById=941160,ctl:ruleRemoveById=941310\\\"\n"
  http-snippet: "modsecurity on;\r\nmodsecurity_rules '\r\n  SecRuleRemoveById 920350\r\n  SecDebugLog /dev/stdout\r\n  SecDebugLogLevel 3\r\n';\r\nmodsecurity_rules_file /etc/nginx/modsecurity/modsecurity.conf;\r\nmodsecurity_rules_file /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf;"
  main-snippet: load_module /etc/nginx/modules/ngx_http_modsecurity_module.so;

Extract from the deployment

spec:  
  template:    
    spec:
      volumes:
        - name: webhook-cert
          secret:
            secretName: ingress-nginx-admission
            defaultMode: 420
        - name: modsecurity-config-custom-default-ruleset-volume
          configMap:
            name: ingress-nginx-controller
            defaultMode: 420
        - name: modsecurity-config-custom-extended-ruleset-volume
          configMap:
            name: ingress-nginx-controller
            defaultMode: 420
        - name: modsecurity-config-custom-rule-removals-volume
          configMap:
            name: ingress-nginx-controller
            defaultMode: 420
      containers:
        - name: controller
          image: >-
            registry.k8s.io/ingress-nginx/controller:v1.10.0@sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c
          args:
            - /nginx-ingress-controller
            - '--publish-service=$(POD_NAMESPACE)/ingress-nginx-controller'
            - '--election-id=ingress-nginx-leader'
            - '--controller-class=k8s.io/ingress-nginx'
            - '--ingress-class=nginx'
            - '--configmap=$(POD_NAMESPACE)/ingress-nginx-controller'
            - '--validating-webhook=:8443'
            - '--validating-webhook-certificate=/usr/local/certificates/cert'
            - '--validating-webhook-key=/usr/local/certificates/key'
            - '--enable-metrics=false'
          resources:
            limits:
              cpu: '1'
              memory: 1Gi
            requests:
              cpu: 100m
              memory: 90Mi
          volumeMounts:
            - name: webhook-cert
              readOnly: true
              mountPath: /usr/local/certificates/
            - name: modsecurity-config-custom-default-ruleset-volume
              mountPath: /etc/modsecurity-rules/default.conf
              subPath: custom-default-ruleset
            - name: modsecurity-config-custom-extended-ruleset-volume
              mountPath: /etc/modsecurity-rules/extended.conf
              subPath: custom-extended-ruleset
            - name: modsecurity-config-custom-rule-removals-volume
              mountPath: /etc/modsecurity-rules/removals.conf
              subPath: custom-rule-removals

That works fine, the files are correctly mounted in the pod.

Now I try to include the different files in the single ingress instance:

nginx.ingress.kubernetes.io/server-snippet: |
      modsecurity on;
      modsecurity_rules_file /etc/modsecurity-rules/default.conf;

As soon as I tried to rollout that, I get the following error:

Error: exit status 1
2024/07/02 07:31:01 [emerg] 2863#2863: "modsecurity_rules_file" directive Rules must have an ID. File: 0x7f791d3180f0 at line: 1
in /tmp/nginx/nginx-cfg2299842162:3548
nginx: [emerg] "modsecurity_rules_file" directive Rules must have an ID. File: 0x7f791d3180f0 at line: 1
in /tmp/nginx/nginx-cfg2299842162:3548
nginx: configuration file /tmp/nginx/nginx-cfg2299842162 test failed

As you notice above, I manually configured modsecurity. I tried that because the standard way using "modsecurity-snippet" did not work as well. It threw the same error, as soon as I added the "Include"-statement.

If I include an existing modsecurity standard config file like /etc/nginx/modsecurity/modsecurity.conf, it works. That's why I assume, that the problem is maybe due to file permissions. The conf-Files I mount from the configmap belong to user root. Well, that's only an assumption, as I tried different variations.

I can rule out problems with my rules. If I put the rules inside the modsecurity-Snippet-Block of the single ingress instance, they work perfectly fine.

ingress-nginx Helm-Chart-Version: 4.10.0

What you expected to happen:

Rules should be loaded and applied for the ingress instance.

NGINX Ingress controller version

NGINX Ingress controller
Release: v1.10.0
Build: 71f78d4
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.25.3

Kubernetes version

v1.28.5

Environment:

How was the ingress-nginx-controller installed:
- If helm was used then please show output of helm ls -A | grep -i ingress

ingress-nginx kube-system 23 2024-06-12 12:10:00.920865595 +0200 CEST deployed ingress-nginx-4.10.0 1.10.0

If helm was used then please show output of helm -n <ingresscontrollernamespace> get values <helmreleasename>

USER-SUPPLIED VALUES:
controller:
  config:
    allow-snippet-annotations: "true"
    custom-default-ruleset: |
      SecRule REQUEST_METHOD \"^HEAD$\" \"id:1001,phase:2,deny,status:403,msg:\'Access denied - HEAD request not allowed\'\"
      SecRule REQUEST_METHOD \"^PATCH$\" \"id:1002,phase:2,deny,status:403,msg:\'Access denied - PATCH$ request not allowed\'\"
      SecRule REQUEST_URI \"@contains wp-admin\" \"id:1003,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@contains wp-content\" \"id:1004,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@contains wp-json\" \"id:1005,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@contains wordpress\" \"id:1006,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@contains LetsEncrypt\" \"id:1007,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@contains cgi-bin\" \"id:1008,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@contains ../\" \"id:1009,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@beginsWith /cgi/\" \"id:1010,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@beginsWith /scripts/\" \"id:1011,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@beginsWith /UI/\" \"id:1012,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@beginsWith /opensso/\" \"id:1013,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@beginsWith /amserver/\" \"id:1014,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@beginsWith /iisprotect/\" \"id:1015,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@beginsWith /NASApp/\" \"id:1016,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@beginsWith /Admin/\" \"id:1017,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@beginsWith /admin/\" \"id:1018,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@beginsWith /apex/\" \"id:1019,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@beginsWith /pls/\" \"id:1020,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@beginsWith /cron/\" \"id:1021,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@beginsWith /~admin/\" \"id:1022,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@beginsWith /~stats/\" \"id:1023,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@beginsWith /~webstats/\" \"id:1024,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@beginsWith /~wsdocs/\" \"id:1025,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@beginsWith /~root/\" \"id:1026,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@beginsWith /~log/\" \"id:1027,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@beginsWith /~1/\" \"id:1028,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@beginsWith /CategoryView.aspx\" \"id:1029,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@beginsWith /ikonboard.cgi\" \"id:1030,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@streq /new\" \"id:1031,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@streq /old\" \"id:1032,phase:2,deny,msg:\'Access is blocked\'\"
      SecRule REQUEST_URI \"@streq /backup\" \"id:1033,phase:2,deny,msg:\'Access is blocked\'\"

      SecRule REQUEST_URI \"!@beginsWith /public/saml\" \"id:1034,phase:2,chain,deny,msg:\'Access is blocked\'\"
        SecRule REQUEST_URI \"@contains .php\"
    custom-extended-ruleset: |-
      SecRule REQUEST_URI \"@beginsWith /login\" \"id:1100,phase:2,chain,deny,msg:\'Invalid login request\'\"
        SecRule REQUEST_URI \"!@rx \/login(\~?[a-z]{0,2}\_?[A-Z]{0,2}\/?)$\" \"id:1101,phase:2,chain\"
          SecRule REQUEST_URI \"!@rx \/login(\~?[a-z]{0,2}\_?[A-Z]{0,2}\/?)\?loggedOut=true$\" \"id:1102,phase:2,chain\"
            SecRule REQUEST_URI \"!@rx \/login(\~?[a-z]{0,2}\_?[A-Z]{0,2}\/?)\?targetUrl=([a-f0-9]*)$\" \"id:1103,phase:2,chain\"
              SecRule REQUEST_URI \"!@rx \/login(\~?[a-z]{0,2}\_?[A-Z]{0,2}\/?)\?authenticationRequired=true\" \"id:1104,phase:2,chain\"
                SecRule REQUEST_URI \"!@rx \/login(\~?[a-z]{0,2}\_?[A-Z]{0,2}\/?)(([\??\&?])(request|execute)=([a-zA-Z]+)\/([a-zA-Z]+)\.([a-z\/]+)){1,2}\"

      SecRule REQUEST_URI \"@beginsWith /regist\" \"id:1200,phase:2,chain,deny,msg:\'Invalid registration request\'\"
        SecRule REQUEST_URI \"!@rx \/(register|registrieren)(\~?[a-z]{0,2}\_?[A-Z]{0,2}\/?)$\" \"id:1201,phase:2,chain\"
          SecRule REQUEST_URI \"!@rx \/(register|registrieren)(\~?[a-z]{0,2}\_?[A-Z]{0,2}\/?)(([\??\&?])(request|execute)=([a-zA-Z]+)\/([a-zA-Z]+)\.([a-z\/]+)){1,2}\" \"id:1203,phase:2,chain\"
            SecRule REQUEST_METHOD \"^GET$\"
    custom-rule-removals: "SecRule REQUEST_URI \\\"@beginsWith /admin?request=Component/Pwa.none&execute=Base/Edit.json/save\\\"
      \\\"id:2000,phase:1,pass,nolog,ctl:ruleRemoveById=941100,ctl:ruleRemoveById=941160,ctl:ruleRemoveById=941310\\\"\nSecRule
      REQUEST_URI \\\"@beginsWith /admin?execute=Editor/H5e.save/w2p/template\\\"
      \\\"id:2001,phase:1,pass,nolog,ctl:ruleRemoveById=941100,ctl:ruleRemoveById=941160,ctl:ruleRemoveById=941310\\\"\t\t\t\nSecRule
      REQUEST_URI \\\"@rx \\/(admin|editorH5e~([a-z]{2})_([A-Z]{2})|editor-h5e\\/)\\?execute=Editor\\/H5e\\.create\\/preview\\/pdf\\\"
      \\\"id:2002,phase:1,pass,nolog,ctl:ruleRemoveById=941100,ctl:ruleRemoveById=941160,ctl:ruleRemoveById=941310\\\"\nSecRule
      REQUEST_URI \\\"@rx \\/(admin|editorH5e~([a-z]{2})_([A-Z]{2})|editor-h5e\\/)\\?execute=Editor\\/H5e\\.update\\\"
      \\\"id:2003,phase:1,pass,nolog,ctl:ruleRemoveById=941100,ctl:ruleRemoveById=941160,ctl:ruleRemoveById=941310\\\"\n"
    enable-modsecurity: "true"
    enable-owasp-modsecurity-crs: "true"
    modsecurity-snippet: |
      SecRuleRemoveById 920350
      SecDebugLog /dev/stdout
      SecDebugLogLevel 3
    variables-hash-bucket-size: "true"
    variables-hash-max-size: "true"
  extraVolumeMounts:
  - mountPath: /tmp/default.conf
    name: modsecurity-config-custom-default-ruleset-volume
    subPath: custom-default-ruleset
  - mountPath: /tmp/extended.conf
    name: modsecurity-config-custom-extended-ruleset-volume
    subPath: custom-extended-ruleset
  - mountPath: /tmp/removals.conf
    name: modsecurity-config-custom-rule-removals-volume
    subPath: custom-rule-removals
  extraVolumes:
  - configMap:
      name: ingress-nginx-controller
    name: modsecurity-config-custom-default-ruleset-volume
  - configMap:
      name: ingress-nginx-controller
    name: modsecurity-config-custom-extended-ruleset-volume
  - configMap:
      name: ingress-nginx-controller
    name: modsecurity-config-custom-rule-removals-volume
  resources:
    limits:
      cpu: 1
      memory: 1Gi

Current State of the controller:
- kubectl describe ingressclasses

Name:         nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.10.0
              helm.sh/chart=ingress-nginx-4.10.0
Annotations:  meta.helm.sh/release-name: ingress-nginx
              meta.helm.sh/release-namespace: kube-system
Controller:   k8s.io/ingress-nginx
Events:       <none>

kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>

Name:             ingress-nginx-controller-5b84f5b5c7-wgx9v
Namespace:        kube-system
Priority:         0
Service Account:  ingress-nginx
Node:             dtm-kkd-w-02/192.168.6.23
Start Time:       Wed, 26 Jun 2024 09:33:21 +0200
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=ingress-nginx
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=ingress-nginx
                  app.kubernetes.io/part-of=ingress-nginx
                  app.kubernetes.io/version=1.10.0
                  helm.sh/chart=ingress-nginx-4.10.0
                  pod-template-hash=5b84f5b5c7
Annotations:      kubectl.kubernetes.io/restartedAt: 2024-06-11T15:59:38Z
Status:           Running
IP:               10.0.2.122
IPs:
  IP:           10.0.2.122
Controlled By:  ReplicaSet/ingress-nginx-controller-5b84f5b5c7
Containers:
  controller:
    Container ID:    containerd://2f257e25be7d3e53640c71336f3a81983159d4cd37274d37dbff80691927dee1
    Image:           registry.k8s.io/ingress-nginx/controller:v1.10.0@sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c
    Image ID:        registry.k8s.io/ingress-nginx/controller@sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c
    Ports:           80/TCP, 443/TCP, 8443/TCP
    Host Ports:      0/TCP, 0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
      --election-id=ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
      --enable-metrics=false
    State:          Running
      Started:      Wed, 26 Jun 2024 10:18:38 +0200
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 26 Jun 2024 10:17:29 +0200
      Finished:     Wed, 26 Jun 2024 10:18:38 +0200
    Ready:          True
    Restart Count:  2
    Limits:
      cpu:     1
      memory:  1Gi
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-5b84f5b5c7-wgx9v (v1:metadata.name)
      POD_NAMESPACE:  kube-system (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /etc/modsecurity-rules/default.conf from modsecurity-config-custom-default-ruleset-volume (rw,path="custom-default-ruleset")
      /etc/modsecurity-rules/extended.conf from modsecurity-config-custom-extended-ruleset-volume (rw,path="custom-extended-ruleset")
      /etc/modsecurity-rules/removals.conf from modsecurity-config-custom-rule-removals-volume (rw,path="custom-rule-removals")
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gkxx2 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  modsecurity-config-custom-default-ruleset-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      ingress-nginx-controller
    Optional:  false
  modsecurity-config-custom-extended-ruleset-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      ingress-nginx-controller
    Optional:  false
  modsecurity-config-custom-rule-removals-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      ingress-nginx-controller
    Optional:  false
  kube-api-access-gkxx2:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>

Name:                     ingress-nginx-controller
Namespace:                kube-system
Labels:                   app.kubernetes.io/component=controller
                          app.kubernetes.io/instance=ingress-nginx
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/part-of=ingress-nginx
                          app.kubernetes.io/version=1.10.0
                          helm.sh/chart=ingress-nginx-4.10.0
                          k8slens-edit-resource-version=v1
Annotations:              meta.helm.sh/release-name: ingress-nginx
                          meta.helm.sh/release-namespace: kube-system
Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.96.136.231
IPs:                      10.96.136.231
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  30080/TCP
Endpoints:                10.0.2.122:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  30443/TCP
Endpoints:                10.0.2.122:443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

kubectl -n <appnamespace> describe ing <ingressname>

Name:             ingress-konmedia-test-stage-brandbox-de
Labels:           app.kubernetes.io/managed-by=Helm
                  helm.toolkit.fluxcd.io/name=brandbox
                  helm.toolkit.fluxcd.io/namespace=konmedia-test
                  k8slens-edit-resource-version=v1
                  type=application
Namespace:        konmedia-test
Address:
Ingress Class:    nginx
Default backend:  <default>
TLS:
  konmedia-test.stage.brandbox.de terminates konmedia-test.stage.brandbox.de
Rules:
  Host                             Path  Backends
  ----                             ----  --------
  konmedia-test.stage.brandbox.de
                                   /   service-application:8080 (10.0.1.15:8080)
Annotations:                       cert-manager.io/cluster-issuer: letsencrypt-prod
                                   meta.helm.sh/release-name: brandbox
                                   meta.helm.sh/release-namespace: konmedia-test
                                   monitoring.brandbox.host/backend-expected-response-code: 401
                                   monitoring.brandbox.host/backend-incident-severity: 3
                                   monitoring.brandbox.host/backend-relative-url: admin
                                   monitoring.brandbox.host/frontend-expected-response-code: 401
                                   monitoring.brandbox.host/frontend-incident-severity: 4
                                   monitoring.brandbox.host/has-frontend: 0
                                   monitoring.brandbox.host/interval: 10m
                                   monitoring.brandbox.host/is-legacy: 0
                                   monitoring.brandbox.host/is-production: 0
                                   monitoring.brandbox.host/monitor: 1
                                   monitoring.brandbox.host/retries: 3
                                   monitoring.brandbox.host/timeout: 3s
                                   nginx.ingress.kubernetes.io/client-body-buffer-size: 100k
                                   nginx.ingress.kubernetes.io/enable-modsecurity: false
                                   nginx.ingress.kubernetes.io/enable-owasp-core-rules: false
                                   nginx.ingress.kubernetes.io/limit-connections: 100
                                   nginx.ingress.kubernetes.io/limit-rps: 100
                                   nginx.ingress.kubernetes.io/modsecurity-snippet:
                                     SecRuleEngine On Include /etc/modsecurity-rules/default.conf Include /etc/modsecurity-rules/extended.conf
                                   nginx.ingress.kubernetes.io/proxy-body-size: 1024M
                                   nginx.ingress.kubernetes.io/proxy-connect-timeout: 10
                                   nginx.ingress.kubernetes.io/proxy-read-timeout: 3600
                                   nginx.ingress.kubernetes.io/proxy-send-timeout: 60
                                   nginx.ingress.kubernetes.io/server-snippet:
                                     modsecurity on;
                                     client_header_timeout 15s;
                                     client_body_timeout 3600s;
                                     ssl_session_tickets off;
                                     reset_timedout_connection on;
                                     keepalive_timeout 65;
                                   nginx.org/client-max-body-size: 1024M
                                   nginx.org/proxy-read-timeout: 3600
Events:                            <none>

How to reproduce this issue:

Install ingress-nginx via Helm-Chart version 4.10.0
Set USER-SUPPLIED-VALUES as defined above
Activate modsecurity and include custom rules file through annotation nginx.ingress.kubernetes.io/modsecurity-snippet

nginx.ingress.kubernetes.io/server-snippet: |
      modsecurity on;
      modsecurity_rules_file /etc/modsecurity-rules/default.conf;

Anything else we need to know:

Investigating the issue, I stumbled upon the following issues:

#8388 > gave me the idea of configuring modsecurity manually
#5673 > gave me the idea of centralizing my rulesets
owasp-modsecurity/ModSecurity-nginx#210 > gave me the information that the snipped length is limited to 4096 bytes, which I can also reproduce

The text was updated successfully, but these errors were encountered:

k8s-ci-robot · 2024-07-02T07:44:20Z

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

longwuyuan · 2024-07-02T08:06:02Z

/remove-kind bug
/kind support

Your issue description is not markdown formatted
The controller pod's kubectl describe is absent
Much of the info asked in a new issue template is absent
You have not specified what document are you following for using custom modsec rules

So fix that issue description and see if you get comments based on useful data (if any)

konmedia-devops · 2024-07-02T08:56:34Z

@longwuyuan Thank you for your feedback. I updated format and content of the issue description.

longwuyuan · 2024-07-02T10:02:51Z

The error visible is

must have an ID. File: 0x7f791d3180f0 at line: 1

does your file have ID ?

konmedia-devops · 2024-07-03T09:08:59Z

@longwuyuan Yes. The error message refers to the rule id, as I understand it. All rules in the file have a rule id, and the same ruleset works, if I put it in the modsecurity-snippet inside the ingress instance. That's why I assume that the file is not readable.

longwuyuan · 2024-07-03T10:03:19Z

If you follow documentation, I can understand what you are trying and work with you.

for example follow these

But this is a absolute requirement

Show the error of insufficient size

Next show the volume mounted but do not use the custom-rules. I want to see the volumes only

And then I too can debug the actual error of not being able to use the custom-rulesets from a volume

I can join a meet.jit.si session. We can sync on slack

konmedia-devops · 2024-07-04T13:02:28Z

@longwuyuan Thank you. I will prepare and get back to you.

longwuyuan · 2024-07-04T13:17:39Z

thank you

longwuyuan · 2024-07-04T13:18:05Z

/triage needs-information

konmedia-devops · 2024-07-16T09:57:01Z

@longwuyuan Thanks for your help. I try to answer your questions beforehand:

https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#enable-modsecurity
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#enable-owasp-modsecurity-crs
And only then come to this one
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#modsecurity-snippet

I followed the documentation and configured modsecurity like explained in these docs.

But this is a absolute requirement

Show the error of insufficient size

Failed to save resource: admission webhook "validate.nginx.ingress.kubernetes.io" denied the request:
-------------------------------------------------------------------------------
Error: exit status 1
2024/07/16 09:47:21 [emerg] 2357#2357: too long parameter, probably missing terminating "'" character in /tmp/nginx/nginx-cfg939531444:3640
nginx: [emerg] too long parameter, probably missing terminating "'" character in /tmp/nginx/nginx-cfg939531444:3640
nginx: configuration file /tmp/nginx/nginx-cfg939531444 test failed

As described in owasp-modsecurity/ModSecurity-nginx#210, nginx has a line limit of 4096 bytes in the configuration file: https://github.com/nginx/nginx/blob/master/src/core/ngx_conf_file.c
As the modsecurity-snippet is include within single quotes, a long snippet exceeds this line length.

Next show the volume mounted but do not use the custom-rules. I want to see the volumes only

...
    spec:
      volumes:
        - name: webhook-cert
          secret:
            secretName: ingress-nginx-admission
            defaultMode: 420
        - name: modsecurity-config-custom-default-ruleset-volume
          configMap:
            name: ingress-nginx-controller
            defaultMode: 420
        - name: modsecurity-config-custom-extended-ruleset-volume
          configMap:
            name: ingress-nginx-controller
            defaultMode: 420
        - name: modsecurity-config-custom-rule-removals-volume
          configMap:
            name: ingress-nginx-controller
            defaultMode: 420
    containers:
      - volumeMounts:
            - name: webhook-cert
              readOnly: true
              mountPath: /usr/local/certificates/
            - name: modsecurity-config-custom-default-ruleset-volume
              mountPath: /etc/modsecurity-rules/default.conf
              subPath: custom-default-ruleset
            - name: modsecurity-config-custom-extended-ruleset-volume
              mountPath: /etc/modsecurity-rules/extended.conf
              subPath: custom-extended-ruleset
            - name: modsecurity-config-custom-rule-removals-volume
              mountPath: /etc/modsecurity-rules/removals.conf
              subPath: custom-rule-removals
...

And then I too can debug the actual error of not being able to use the custom-rulesets from a volume

I can join a meet.jit.si session. We can sync on slack

Great. How can I reach you?

longwuyuan · 2024-07-16T11:39:33Z

I am on Kubernetes Slack in the ingress-nginx-users channel

konmedia-devops · 2024-07-16T13:35:25Z

@longwuyuan

I am on Kubernetes Slack in the ingress-nginx-users channel

I need an invitation, right? Sry, I am not familiar with Slack...

longwuyuan · 2024-07-16T13:37:47Z

channel ingress-nginx
slack kubernetes.slack.com
registration slack.kubernetes.io

konmedia-devops · 2024-07-22T12:57:18Z

Problem could be resolved. My rulesets still contained the escaping \ for the quotation marks, that are necessary in the modsecurity-snippet.

konmedia-devops added the kind/bug Categorizes issue or PR as related to a bug. label Jul 2, 2024

k8s-ci-robot added needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority labels Jul 2, 2024

strongjz added this to [SIG Network] Ingress NGINX Jul 2, 2024

k8s-ci-robot added kind/support Categorizes issue or PR as a support question. and removed kind/bug Categorizes issue or PR as related to a bug. labels Jul 2, 2024

k8s-ci-robot added the triage/needs-information Indicates an issue needs more information in order to work on it. label Jul 4, 2024

konmedia-devops closed this as completed Jul 22, 2024

github-project-automation bot moved this to Done in [SIG Network] Ingress NGINX Jul 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

modsecurity: Including custom rules file throws error #11531

modsecurity: Including custom rules file throws error #11531

konmedia-devops commented Jul 2, 2024 •

edited

Loading

k8s-ci-robot commented Jul 2, 2024

longwuyuan commented Jul 2, 2024

konmedia-devops commented Jul 2, 2024

longwuyuan commented Jul 2, 2024

konmedia-devops commented Jul 3, 2024

longwuyuan commented Jul 3, 2024

konmedia-devops commented Jul 4, 2024

longwuyuan commented Jul 4, 2024

longwuyuan commented Jul 4, 2024

konmedia-devops commented Jul 16, 2024

longwuyuan commented Jul 16, 2024

konmedia-devops commented Jul 16, 2024

longwuyuan commented Jul 16, 2024

konmedia-devops commented Jul 22, 2024

modsecurity: Including custom rules file throws error #11531

modsecurity: Including custom rules file throws error #11531

Comments

konmedia-devops commented Jul 2, 2024 • edited Loading

k8s-ci-robot commented Jul 2, 2024

longwuyuan commented Jul 2, 2024

konmedia-devops commented Jul 2, 2024

longwuyuan commented Jul 2, 2024

konmedia-devops commented Jul 3, 2024

longwuyuan commented Jul 3, 2024

konmedia-devops commented Jul 4, 2024

longwuyuan commented Jul 4, 2024

longwuyuan commented Jul 4, 2024

konmedia-devops commented Jul 16, 2024

longwuyuan commented Jul 16, 2024

konmedia-devops commented Jul 16, 2024

longwuyuan commented Jul 16, 2024

konmedia-devops commented Jul 22, 2024

konmedia-devops commented Jul 2, 2024 •

edited

Loading