applyFSGroup happens regardless of fsGroupChangePolicy

[flmmartins]

Hello,

What happened:

I have a NFS share with following permissions:

drwxrwx--- talos talos

I created vault pod with the following securityContext. My goal was to make very tight permissions in NFS due to security.

securityContext:
│     fsGroup: 3001
│     runAsGroup: 3001
│     runAsUser: 3001 #vault

I install it. Everything works as expected and you can see files are created with the exact group and user I wanted:

$ls -la pvc-...
drwxrwxr-x talos vault .
drwx------ vault vault raft

Once I restart the pod. I get the error:

42c7e" : applyFSGroup failed for vol SERVER_IP#mnt/default/talos#pvc-b8335f99-53a9-4e90-9cb5-929e3bb42c7e##: open /var/lib/kubelet/pods/ffa5bff2-02a1-4aad-a410-a2d812cd34d5/volumes/kubernetes.io~csi/pvc-b8335f99-53a9-4e90-9cb5-929e3bb42c7e/mount/raft: permission denied

Then I added fsGroupChangePolicy: "OnRootMismatch" so now we have:

securityContext:
      runAsUser: 3001 #vault user
      runAsGroup: 3001 #vault group
      fsGroup: 3001
      fsGroupChangePolicy: "OnRootMismatch"

Same error happens.

What you expected to happen:

I expected that due to the fsGroupChangePolicy: "OnRootMismatch" it would only try to change the fs permissions on first time when the permissions would not match fsGroup. Following mounts would not try to change the permissions of files created by the application.

I solved the issue by defining fsGroup as talos. But... that is not exactly secure bc the application/pod could set files with 750 for example and that would allow the group outside vault to read application information.

How to reproduce it:

Install hashicorp vault with raft enabled. Set security context to the above. Restart the pod

Environment:

• CSI Driver version: v5.0.2
• Kubernetes version (use kubectl version): v1.31.2
• OS (e.g. from /etc/os-release): "Debian GNU/Linux"

[andyzhangx]

what is the csi driver logs on the node? follow by: https://github.com/kubernetes-csi/csi-driver-nfs/blob/master/docs/csi-debug.md#case2-volume-mountunmount-failed

[flmmartins]

Hello, thanks for answering!

csi-nfs-node-description-for-issue.log

csi-nfs-node-pvc-222c87df-f329-41e4-8ef1-a0bcefd6bf72.log