Image update automation not committing the resolved version to git

[sanjvij]

Hi team,

Thanks for your help so far. I am stuck at implementing a use case where by the image update automation policy is not applying the changes to the git.

When I run the command as below, I can see that flux was able to detect a new version in the registry but never committed the same to git.
(base) sanj@Sanjs-Air app-cluster % flux get image policy staging
NAME READY MESSAGE LATEST IMAGE
staging True Latest image tag for 'sanjvij01/getting-started' resolved to: v3.0.2 sanjvij01/getting-started:v3.0.2

(base) sanj@Sanjs-Air app-cluster % kubectl get deployment/getting-started-image -n staging -oyaml | grep 'image'
name: getting-started-image
selfLink: /apis/apps/v1/namespaces/staging/deployments/getting-started-image
- image: sanjvij01/getting-started:v3.0.1
imagePullPolicy: IfNotPresent
message: ReplicaSet "getting-started-image-554964548d" has successfully progressed.

my image update automation file looks like below. I have a feeling I have done something in which case feel free to point.

apiVersion: image.toolkit.fluxcd.io/v1alpha1
kind: ImageUpdateAutomation
metadata:
name: flux-system
namespace: flux-system
spec:
checkout:
branch: master
gitRepositoryRef:
name: flux-staging
commit:
authorEmail: sanjvij01@gmail.com
authorName: sanjvij
messageTemplate: '{{range .Updated.Images}}{{println .}}{{end}}'
interval: 1m0s
push:
branch: master
update:
path: ./
strategy: Setters

Let me know if you need me to provide any further info.

[rayterrill]

I'm running into a similar issue. Is there a way to debug the image-automation-controller to get more info on why we're getting "no updates made"?

I've got two clusters:

• 1.19 in EKS, looks like it's running v0.11.0 (PROD)
• 1.20 as a K3s cluster, running v0.12.3. (TEST)

I may try to downgrade my TEST cluster to v0.11.0 to see if that lets things work.

[rayterrill]

Downgraded cluster to v0.11.0, still not able to get image-automation-controller working correctly, and not sure where to start looking to understand why it isn't making an update.

[rayterrill]

Update - Dug into the manifests for the flux components - looks like we can put the image-automation-controller in debug mode:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/instance: flux-system
    app.kubernetes.io/version: v0.11.0
    control-plane: controller
  name: image-automation-controller
  namespace: flux-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: image-automation-controller
  template:
    metadata:
      annotations:
        prometheus.io/port: "8080"
        prometheus.io/scrape: "true"
      labels:
        app: image-automation-controller
    spec:
      containers:
      - args:
        - --events-addr=http://notification-controller/
        - --watch-all-namespaces=true
        - --log-level=debug
        - --log-encoding=json
        - --enable-leader-election

[rayterrill]

Getting:

no changes made in working directory; no commit

[rayterrill]

Played around with the yaml for my definition - maybe something was goofed up with the imagepolicy comment? I eventually got it working but it would really be nice to have some additional mechanisms to figure out why this wasn't working.

[bobrossthepainter]

Hi @rayterrill,
I just came across a similar issue today. Could it be the case that this issue is connected to a directory name, where the yaml file containing the imagePolicy reference is located?
E.g. for me the update works when the imagePolicy reference is located in:
deploy/overlays/int/trololo/patch.yaml
on the other hand it is not working if its located in here:
deploy/overlays/int/user-ui-config-service/patch.yaml

BR
Robert

[bobrossthepainter]

I just found out, that the problem is indeed related to the directory where the file holding the imagePolicy ref is located.
The alphabetically ascending last directory in a parent directory is ignored by the imagePolicy resolving algorithm.

My fix was adding another dir zzz with a .gitkeep file inside and it magically worked. :D

[rayterrill]

Potentially, yes. I did end up moving my stuff around in directories - so maybe that helped resolve it.

Really wish there was a way for DEBUG mode to give enough detail to determine why it's not working so we can self-resolve these kinds of issues.

[squaremo]

@rayterrill What sort of debug output would have helped you?

[rayterrill]

Ideally some way to understand why it didn't work - something like "nothing to update" or even better something like "image would be updated but setters didn't match anything" - something to indicate the reconciliation loop found work to do but something in the config prevented it from being written. I believe my problem was indeed that my folder structure and setters definition were not aligned (backed into this by moving things around until I discovered that was the problem).

[risalfalah]

Please correct me if I'm wrong.

The GitRepository, ImageRepository, ImagePolicy, and ImageUpdateAutomation must be in the same namespace. Then you need to add the image policy marker. eg {"$imagepolicy": "<policy-namespace>:<policy-name>"}. Meaning when you have multiple namespace, the solutions are

1. Create GitRepository, ImageRepository, ImagePolicy, ImageUpdateAutomation in flux-system namespace. Then the image policy marker points to flux-system:<policy_name>
2. Create GitRepository in every namespace that want to have ImageUpdateAutomation.

I don't know if it is expected by the design or not. I already tested the first approach and it works well. This is the fact that i found.

• ImageUpdateAutomation can only refer to GitRepository in same namespace (1) (2)
• ImageUpdateAutomation only get ImagePolicy in same namespace (1)

[squaremo]

Ideally some way to understand why it didn't work - something like "nothing to update" or even better something like "image would be updated but setters didn't match anything" - something to indicate the reconciliation loop found work to do but something in the config prevented it from being written.

The controller argument --log-level=debug now results in lots of tracing output: #190. (This will be moved to --log-level=trace at some point soon).