ROX-30064: Script to generate catalog-template.yaml #137

kurlov · 2025-07-01T12:28:26Z

Adds a new Go script to generates catalog-template.yaml from a new bundles.yaml file.
The bundles.yaml has a lists operator bundle images with versions, the oldest_supported_version and a list of broken_versions.
- oldest_supported_version specifies what is the lowest supported version. Any version or channel before oldest_supported_version will be marked as deprecated.
- broken_versions a list of versions which should be skipped. For each broken version X.Y.Z the script adds "skips" for all versions > X.Y.Z and < X.Y+2.0
Add CI step to checks which validates that catalog-template.yaml is up-to-date with bundles.yaml
Add CI to run go unit tests if cmd folder is changed

catalog-template.yaml changes:

Now it's autogenerated
All yaml anchors are dropped
Add schema: olm.bundle deprecation references for all version < oldest_supported_version. So not only channels are deprecated.
Add rhacs-3.63 channel
skips is only added for broken version + 2 minor version
Channels keeps all previous version within the same major version (e.g. 3.66 channel has all version <= 3.66.x)
latest channel has all 3.X.X versions
stable channel has all 4.X.X versions

Testing this PR catalogs on OCP 4.19:

image: quay.io/rhacs-eng/stackrox-operator-index@sha256:9345ef4e16b463205ab9dcf4446c5a34babc8c497ad9cbbeae327fb44b2f356d
commit: ca7b4bd

Now not only channels but versions are also deprecated.

Test locally:

make generate-catalog-template
make clean && make valid-catalogs

…-structure

generate.go

msugakov

Scratched the surface a bit. Got enough comments for the first round of review.
Please expect more review rounds and more comments.

generate.go

.gitignore

…-structure

msugakov

Comments on inter-diff.

cmd/generate-catalog/generate.go

cmd/generate-catalog/types.go

cmd/generate-catalog/generate.go

msugakov

A couple more thoughts for today. Will make another pass after you update the PR.

cmd/generate-catalog/generate.go

msugakov · 2025-08-18T16:55:56Z

cmd/generate-catalog/generate.go

+	validForChannel := channel.YStreamVersion != nil &&
+		entry.Version.Major() == channel.YStreamVersion.Major() &&
+		entry.Version.Minor() <= channel.YStreamVersion.Minor()


This must be consistent with what addReplaces does (ref. versionsWithoutReplaces). Right now it is not if you consider what happens when we'd introduce 5.0.0. Do you have ideas how to make these two parts more consistent and at least make sure that when we change one we don't forget the other?

I at this point don't have a suggestion. Need to think about it.

I wasn't able to find a nice solution, probably will come with some idea tomorrow

If we add 5.0.0 to versionsWithoutReplaces that means that we want to introduce yet another non versioned channel (like stable and latest). So it's hard to implement a nice solution unless we add a new configuration to bundles.yaml (e.g.:

... pivot_channels: - name: latest start_version: 3.62.0 - name: stable start_version: 4.0.0 - name: new start_version: 5.0.0 ...

I think it's fine to keep the same deprecation messages for deprecated non versioned (pivotal) channels

I don't believe there are plans to do that: introduce another rolling channel for 5.0.0+. stable will remain and will get new versions.

bundles.yaml

cmd/generate-catalog/generate.go

Co-authored-by: Misha Sugakov <[email protected]>

msugakov

Quick incomplete check.

cmd/generate-catalog/types.go

cmd/generate-catalog/generate.go

msugakov · 2025-08-20T15:54:07Z

cmd/generate-catalog/generate.go

 		return fmt.Errorf("failed to read bundle list file: %v", err)
 	}
-	versions := getAllVersions(input.Images)
+	versions := getAllVersions(config.Images)


Sorry, I don't understand your reply.

If it is more convenient for some functions to work with []*semver.Version instead of []InputBundleImage (and I agree it is), provide them with []*semver.Version.

My concern is that the line versions := getAllVersions(config.Images) is quite low-level in this high-level generateCatalogTemplateFile() function so that it makes me feel uncomfortable about it.

My suggestion is to push it lower in the stack: either to declare Versions []*semver.Version attribute as part of the Input struct and populate that in the readInputFile() call, or declare func (i *Input) Versions() []*semver.Version { ... } to construct that slice dynamically.
Whichever way, the line versions := getAllVersions(config.Images) will disappear from generateCatalogTemplateFile() and that's what I advocate for.

cmd/generate-catalog/generate.go

bundles.yaml

msugakov

Made another incomplete pass. I still skip unit tests.

cmd/generate-catalog/generate.go

cmd/generate-catalog/types.go

cmd/generate-catalog/generate.go

cmd/generate-catalog/types.go

cmd/generate-catalog/generate.go

Co-authored-by: Misha Sugakov <[email protected]>

msugakov

Some things I spotted in the inter-diff

cmd/generate-catalog/types.go

cmd/generate-catalog/generate.go

msugakov · 2025-09-01T16:51:59Z

cmd/generate-catalog/generate.go

+	for i, channel := range channels {
+		for _, entry := range channelEntries {
+			if channelShouldHaveEntry(channel, entry) {
+				channels[i].Entries = append(channels[i].Entries, entry)


There is a mutation of the original channel list.

I can't spot it. Please show it to me.

I see a mutation of .Entries in blah.Entries; blah stays the same, doesn't it?

cmd/generate-catalog/types.go

msugakov · 2025-09-01T17:09:43Z

cmd/generate-catalog/generate_test.go

+func TestValidateImageReferences(t *testing.T) {
+	tests := []struct {


I can suggest a couple more cases. Just providing the references to test:

bare-image-name-without-registry@sha256:6cdcf20771f9c46640b466f804190d00eaf2e59caee6d420436e78b283d177bf

example.com/image:my-fancy-tag-v1@sha256:6cdcf20771f9c46640b466f804190d00eaf2e59caee6d420436e78b283d177bf

example.com/image@md5:9241e37fcf7f3f88c5e944bd46b0a268

example.com/image@sha256:0123456789

I changed the library for image reference validation to make it possible to validate repository. I added those cases except for case number 2. Do you mean that it should fail? This tag is valid according to specification

cmd/generate-catalog/generate.go

cmd/generate-catalog/generate_test.go

bundles.yaml

WIP script to generate catalog-template.yaml

f7abddb

This comment was marked as off-topic.

Sign in to view

kurlov changed the title ~~WIP script to generate catalog-template.yaml~~ DO NOT REVIEW YET: script to generate catalog-template.yaml Jul 1, 2025

kurlov changed the title ~~DO NOT REVIEW YET: script to generate catalog-template.yaml~~ WIP: script to generate catalog-template.yaml Jul 1, 2025

kurlov and others added 7 commits July 7, 2025 06:06

Drop previous approach

9077e29

Remove redundant bash script

7f45122

Improve

fb74d4e

Fix generation bugs

67f98d8

Use latest yaml library

da2882f

Merge branch 'master' into akurlov/add-catalog-generation-from-folder…

52ecdd5

…-structure

Merge branch 'master' into akurlov/add-catalog-generation-from-folder…

cacdf62

…-structure

kurlov changed the title ~~WIP: script to generate catalog-template.yaml~~ Script to generate catalog-template.yaml Jul 9, 2025

kurlov added 5 commits July 9, 2025 11:37

Drop unused structs and rename existing

742e929

Simplify file writting

9f7710d

use semver library instead

38543b2

Add tests

d8a4da6

Add additional comment and replace original catalog-temlate

09226f7

kurlov marked this pull request as ready for review July 10, 2025 12:23

kurlov commented Jul 10, 2025

View reviewed changes

generate.go Outdated Show resolved Hide resolved

kurlov changed the title ~~Script to generate catalog-template.yaml~~ ROX-30064: Script to generate catalog-template.yaml Jul 10, 2025

kurlov requested review from mclasmeier and porridge July 11, 2025 12:57

msugakov self-requested a review July 11, 2025 13:46

msugakov reviewed Jul 11, 2025

View reviewed changes

kurlov and others added 3 commits July 14, 2025 15:15

Merge branch 'master' into akurlov/add-catalog-generation-from-folder…

f76395d

…-structure

Move files and address review comments

b057e1a

Use Parse for validating image reference

9a6e92f

kurlov requested a review from msugakov July 15, 2025 10:05

kurlov and others added 2 commits July 15, 2025 13:49

Merge branch 'master' into akurlov/add-catalog-generation-from-folder…

237dbe6

…-structure

Rollback catalog-template

31a607a

Review comments

4454f14

kurlov requested a review from msugakov August 18, 2025 14:32

msugakov reviewed Aug 18, 2025

View reviewed changes

cmd/generate-catalog/generate.go Outdated Show resolved Hide resolved

kurlov and others added 4 commits August 19, 2025 16:38

Apply suggestions from code review

08deccb

Co-authored-by: Misha Sugakov <[email protected]>

Review comments

259c9ce

Use StrictNewVersion

b9850bf

Drop redundant patch == 0

87ba005

kurlov requested a review from msugakov August 19, 2025 17:28

Resolve conflicts

fbe2c74

msugakov reviewed Aug 20, 2025

View reviewed changes

msugakov reviewed Aug 21, 2025

View reviewed changes

kurlov and others added 8 commits August 21, 2025 15:48

Review comments

9930683

Add TODO

44a2530

Update catalogs

d5b08cb

Add brokenVersionSkippingOffset constant

cc03d50

Apply suggestions from code review

4ae9576

Co-authored-by: Misha Sugakov <[email protected]>

Fix compiling

5c47c24

Review comments

c7673cb

Review comments

aaa6b2f

kurlov requested a review from msugakov August 21, 2025 15:28

Fix newline

0aff37c

msugakov reviewed Sep 2, 2025

View reviewed changes

bundles.yaml Outdated Show resolved Hide resolved

bundles.yaml Show resolved Hide resolved

kurlov added 4 commits September 16, 2025 14:05

Add 4.8.3 and 4.8.4

c1c9931

Change image reference validation

70d38c4

Fix uunittest

99fbb14

Resolve conflicts

7a32153

kurlov requested a review from msugakov September 17, 2025 15:25

		func TestValidateImageReferences(t *testing.T) {
		tests := []struct {

ROX-30064: Script to generate catalog-template.yaml #137

Are you sure you want to change the base?

ROX-30064: Script to generate catalog-template.yaml #137

Uh oh!

Conversation

kurlov commented Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

catalog-template.yaml changes:

Testing this PR catalogs on OCP 4.19:

Test locally:

Uh oh!

This comment was marked as off-topic.

Uh oh!

msugakov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

msugakov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

msugakov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

msugakov Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

kurlov Aug 19, 2025

Choose a reason for hiding this comment

Uh oh!

kurlov Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

msugakov Aug 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

msugakov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

msugakov Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

msugakov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kurlov commented Jul 1, 2025 •

edited

Loading

kurlov Aug 20, 2025 •

edited

Loading