[ci] stage images with final version during external release pipelines

[cdesiniotis]

This change ensures that images staged for external release get tagged with the release version, e.g. x.y.z, and not a commit sha.

[tariq1890]

LGTM. Needs rebase

[elezar]

I don't think this is required. Why don't we ONLY use SHAs in staging? We would then promote SHA-tagged images to prod (re-versioning) then.

[tariq1890]

I think pushing tags to staging makes it easier to track images. SHAs are difficult to read

[cdesiniotis]

With the new publishing flow, we need to register images in nspect before release. I think I would prefer if we registered release tags rather than SHAs.

[elezar]

I'm not going to fight too hard on this, but I don't want to underestimate the value consistency. Up to now, we've always used a SHA for unreleased images and only tagged the images in the final release process. I understand that this is unwieldly in nspect, but could be made a lot simpler with automation (I know that nspect does allow SHAs as "versions").

Also, since we are now pushing to staging in our internal pipelines, pushing a tag only needs to trigger the release steps:

1. Update nspect
2. Create a LaunchAPI MR

Is it possible to push both?

[cdesiniotis]

I am open to discussing this further before moving forward with this PR. I will make another iteration on my document to cover these details.

Is it possible to push both?

Could you clarify what you are asking here?

[elezar]

Could you clarify what you are asking here?

I mean could we reach a state where we have both a SHA-tagged image and a version-taged image in the staging repository? (I assume that this is already the case since this pipeline runs for both the repo sync and a tag push).

[tariq1890]

Hi @elezar , I disagree that we should push SHA-tagged images to a staging repository. The purpose of the staging repository is to push the images that are staged for production release. If we stick to this convention, it becomes a lot easier to track the history of images. And if we were to push SHA-tagged images to staging, the number of images in the staging repository would grow over time. The sha-tagged images represent our daily-builds and they should be published in a different registry IMO

[elezar]

@tariq1890 how do we handle multiple rc's in this case? Let us say we tag some sha with v1.2.3 in preparation for the release. This pushes the image to staging with the image tag v1.2.3. In final scanning / testing we discover that a change is required. How do we handle this? Do we force-push the tag v1.2.3? How do we keep track that we are testing the update v1.2.3 image and not the previous one.

We use a short-SHA git reference for images and build artifacts because these uniquely identify a particular state of the codebase. We also only build these images once in public and all other operations are essentially moving these around.

If our concern is that we are going to fill up some storage, that is a different problem and one that can be addressed in a number of ways. Two that I can think of are:

1. Some garbage collection mechanism for old images in the staging registry.
2. Only triggering the push of a SHA-tagged image to the staging registry on some other signal.\

Note that any branches in a pipeline that are only triggered occasionally have a higher risk of breaking precisely at the point in time when they are critical. Ensuring that as much of the release process is covered by every commit ensures that we detect problems sooner. One example where we had such problems in the past is container scans -- which were only triggered on release.