Node collector scan job error on AKS agent nodes

[dergyitheron]

What steps did you take and what happened:

I deployed trivy-operator via Helm with default values on multiple AKS clusters. Vulnerability scans are running for images without any issues, I can collect metrics about it with prometheus.

There is an error produced by node collector reappearing for each node in the AKS cluster. This is happening for every AKS cluster I have the trivy-operator on.

All those mentioned nodes are regular agent nodes.

{"level":"error","ts":"2025-03-13T16:10:24Z","msg":"Reconciler error","controller":"node","controllerGroup":"","controllerKind":"Node","Node":{"name":"aks-companypool-15499263-vmss00001l"},"namespace":"","name":"aks-companypool-15499263-vmss00001l","reconcileID":"a3120cb0-ecf8-42c5-82b3-4dfb83925b75","error":"creating job: no compliance commands found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:224"}
{"level":"error","ts":"2025-03-13T16:10:54Z","msg":"Reconciler error","controller":"node","controllerGroup":"","controllerKind":"Node","Node":{"name":"aks-companypool-15499263-vmss00000f"},"namespace":"","name":"aks-companypool-15499263-vmss00000f","reconcileID":"51f77a5f-b3a3-4cbc-bb26-c96c8621fd87","error":"creating job: no compliance commands found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:224"}
{"level":"error","ts":"2025-03-13T16:11:02Z","msg":"Reconciler error","controller":"node","controllerGroup":"","controllerKind":"Node","Node":{"name":"aks-companypool-15499263-vmss00001g"},"namespace":"","name":"aks-companypool-15499263-vmss00001g","reconcileID":"daafd590-b69d-4cfd-bc83-973df0248397","error":"creating job: no compliance commands found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:224"}
{"level":"error","ts":"2025-03-13T16:11:30Z","msg":"Reconciler error","controller":"node","controllerGroup":"","controllerKind":"Node","Node":{"name":"aks-companypool-15499263-vmss00000x"},"namespace":"","name":"aks-companypool-15499263-vmss00000x","reconcileID":"5336eb7e-72f2-462c-b500-e86c3cc0b5a2","error":"creating job: no compliance commands found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.1/pkg/internal/controller/controller.go:224"}

What could be the root cause of this?

What did you expect to happen:

The error message should either be more verbose telling me what exactly the issue is, maybe it's justified by I have no clue what is causing it from this error message.

Anything else you would like to add:

This issue is similar but it mentions solution for the scanner not being able to scan controllplane nodes, my nodes are agent nodes. #2202 (comment)

I can observe this across tenants, subscriptions and AKS instances.

Environment:

• Trivy-Operator version (use trivy-operator version):

{"Version":"0.23.0","Commit":"72ac532ab1a0d77a0a63131cf08392809ed24b6c","Date":"2024-11-26T09:02:15Z","Executable":""}

Helmchart version 0.25.0

• Kubernetes version (use kubectl version):

Client Version: v1.32.2
Kustomize Version: v5.5.0
Server Version: v1.31.5

• OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc): Ubuntu 22.04.5 LTS

[afdesk]

hi @dergyitheron! thanks for the report!

Unfortunately, Trivy operator makes no guarantees on managed Kubernetes flavors, so it's quite possible we don't fully support it.

but I have some ideas.

"creating job: no compliance commands found" means Trivy-operator can't run scan jobs, it seems there are no credentials for it.

Could you re-check that creadentials are granted at first? thanks

[dergyitheron]

Hi, thank you for your response.

I'm not sure I understand what you mean. What credentials are needed for node scanning?

Image scanning when new resources are deployed to the AKS cluster run just fine. As I understand the error it has issues with scanning nodes. In case I don't need the node scanning is there a way to turn it off?

[Connor-Wood]

Also using AKS and seeing this when upgrading trivy operator helm chart from 0.24.1 to 0.25.0... very consistent behavior but not seeing any reason for it from the changelog... will downgrade for now

[dergyitheron]

This is my workaround for now to at least stop it from logging the errors.

nodeCollector:
  excludeNodes: kubernetes.azure.com/role=agent

Would like to solve it out in more elegant way, or just make it work if possible.

[afdesk]

it's a known situation for managed kubernetes
there are details and solution here: #2202 (comment)