Automation for updating templates

[AkihiroSuda]

It is really hard for me to create a PR like #1236 to update the template image digests.

We have to have a tool for updating these templates automatically.
The tool must retain comment lines and indentation styles in the YAMLs.

---

Frequently updated images:

• Ubuntu Add hack/update-template-ubuntu.sh #2702
• Debian Refactor update-template-ubuntu.sh and add update-template-debian.sh #2731
• ArchLinux Add hack/update-template-archlinux.sh #2788
• CentOS Stream Add hack/update-template-centos-stream.sh #2794

Less frequent ones:

• Alpine Add hack/update-template-alpine.sh #2811
• AlmaLinux Add hack/update-template-almalinux.sh #2796
• Rocky Linux Add hack/update-template-rocky.sh #2806
• Oracle Linux Add hack/update-template-oraclelinux.sh #2820
• Fedora Add hack/update-template-fedora.sh #2826
• openSUSE Leap Add hack/update-template-opensuse.sh #2852

[afbjorklund]

Maybe also some means of sharing them, with some kind of FROM system ?

• Inherit system images and share them between examples #824

[AkihiroSuda]

Maybe also some means of sharing them, with some kind of FROM system ?

• Inherit system images and share them between examples #824

Yes, but that is a separate issue

[jlm0x017]

Do you already have the link to the updated image? Or is that step needed as well?
I expect you grab the checksum provided at the source, rather than generating it yourself, but please do confirm.

[AkihiroSuda]

Do you already have the link to the updated image?

No, e.g., we have to detect the latest version 20230124-1270 from https://cloud.debian.org/images/cloud/bullseye/ , but I'm not sure what is the robust way to do this.

w3m | grep might be enough, but seriously we should also consider adopting some machine learning stuff.

[afbjorklund]

If I remember correctly, there was some standard metadata for scraping upstream for new releases to package...

Like https://wiki.debian.org/debian/watch

Maybe something like that can be used here, to "describe" the various vendors and where they put their binaries ?

Some thing simple, with place holders for date strings and checksums.

[afbjorklund]

Example output:

https://qa.debian.org/cgi-bin/watch?pkg=containerd

Unfortunately, the others are not available as packages. But maybe something similar to this, but for images:

https://repology.org/project/nerdctl/versions

[lobshunter]

What about a naive bash/python script to replace placeholder strings in yaml? Something like:

ubuntu_image = `w3m | grep -e "xxx"`
sed -i "s/UBUNTU_IMAGE/$ubuntu_image/g" ubuntu.yaml

The robustness solely relies on consistent file naming of upstreams(counting on hyrum's law). Sadly I didn't find any public tool to retrive latest released images.

[AkihiroSuda]

I'm now planning to use yq

[balajiv113]

How about using libosinfo (osinfo-db os) ??

I could see their db has info of different os variants
https://gitlab.com/libosinfo/osinfo-db/-/tree/main/data/os

We might need to write a python wrapper on top of this library (the tool osinfo-db is not giving out info on image download URL's by architecture that is present in the xml file)

[afbjorklund]

yq can read xml too

yq -p xml -P