Skip to content

[ci] stage images with final version during external release pipelines #1232

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[ci] stage images with final version during external release pipelines #1232

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .common-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,10 @@ scan-ubi9-arm64:
OUT_REGISTRY_TOKEN: "${NGC_REGISTRY_TOKEN}"
OUT_REGISTRY: "${NGC_REGISTRY}"
OUT_IMAGE_NAME: "${NGC_REGISTRY_STAGING_IMAGE_NAME}"
rules:
- if: $CI_COMMIT_TAG
variables:
OUT_IMAGE_VERSION: "${CI_COMMIT_TAG}"
Comment on lines +193 to +196
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is required. Why don't we ONLY use SHAs in staging? We would then promote SHA-tagged images to prod (re-versioning) then.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think pushing tags to staging makes it easier to track images. SHAs are difficult to read

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the new publishing flow, we need to register images in nspect before release. I think I would prefer if we registered release tags rather than SHAs.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not going to fight too hard on this, but I don't want to underestimate the value consistency. Up to now, we've always used a SHA for unreleased images and only tagged the images in the final release process. I understand that this is unwieldly in nspect, but could be made a lot simpler with automation (I know that nspect does allow SHAs as "versions").

Also, since we are now pushing to staging in our internal pipelines, pushing a tag only needs to trigger the release steps:

  1. Update nspect
  2. Create a LaunchAPI MR

Is it possible to push both?

Copy link
Contributor Author

@cdesiniotis cdesiniotis Apr 16, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am open to discussing this further before moving forward with this PR. I will make another iteration on my document to cover these details.

Is it possible to push both?

Could you clarify what you are asking here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you clarify what you are asking here?

I mean could we reach a state where we have both a SHA-tagged image and a version-taged image in the staging repository? (I assume that this is already the case since this pipeline runs for both the repo sync and a tag push).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @elezar , I disagree that we should push SHA-tagged images to a staging repository. The purpose of the staging repository is to push the images that are staged for production release. If we stick to this convention, it becomes a lot easier to track the history of images. And if we were to push SHA-tagged images to staging, the number of images in the staging repository would grow over time. The sha-tagged images represent our daily-builds and they should be published in a different registry IMO

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tariq1890 how do we handle multiple rc's in this case? Let us say we tag some sha with v1.2.3 in preparation for the release. This pushes the image to staging with the image tag v1.2.3. In final scanning / testing we discover that a change is required. How do we handle this? Do we force-push the tag v1.2.3? How do we keep track that we are testing the update v1.2.3 image and not the previous one.

We use a short-SHA git reference for images and build artifacts because these uniquely identify a particular state of the codebase. We also only build these images once in public and all other operations are essentially moving these around.

If our concern is that we are going to fill up some storage, that is a different problem and one that can be addressed in a number of ways. Two that I can think of are:

  1. Some garbage collection mechanism for old images in the staging registry.
  2. Only triggering the push of a SHA-tagged image to the staging registry on some other signal.\

Note that any branches in a pipeline that are only triggered occasionally have a higher risk of breaking precisely at the point in time when they are critical. Ensuring that as much of the release process is covered by every commit ensures that we detect problems sooner. One example where we had such problems in the past is container scans -- which were only triggered on release.


# Define an external release step that pushes an image to an external repository.
# This includes a devlopment image off main.
Expand Down