Skip to content

NO-JIRA: Updates CI images to use RHEL10 for early feedback #89

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bharath-b-rh wants to merge 1 commit into
base: main
Choose a base branch
from
Open

NO-JIRA: Updates CI images to use RHEL10 for early feedback #89

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion images/ci/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ COPY . .

RUN go build -tags $GO_BUILD_TAGS -o external-secrets-operator cmd/external-secrets-operator/main.go

FROM registry.access.redhat.com/ubi9-minimal:9.4
FROM registry.ci.openshift.org/rhcos-devel/ocp-4.21-10.1:4.21.0-ec.3-x86_64
Copy link

@coderabbitai coderabbitai bot Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

# First, let's examine the Dockerfile and surrounding context
cat -n images/ci/Dockerfile | head -30

Repository: openshift/external-secrets-operator

Length of output: 214

🏁 Script executed:

# Check if there are other Dockerfiles using similar versioning patterns
find . -name "Dockerfile*" -type f | head -20

Repository: openshift/external-secrets-operator

Length of output: 214

🏁 Script executed:

# Search for other base image references to understand versioning conventions used
rg "registry\.ci\.openshift\.org" --type dockerfile -A 2 -B 2

Repository: openshift/external-secrets-operator

Length of output: 214

🏁 Script executed:

# Look for CI configuration or documentation about image stability
find . -type f \( -name "*.md" -o -name "*.txt" -o -name ".gitlab-ci.yml" -o -name "Jenkinsfile" -o -name ".github" \) | grep -E "(README|CI|pipeline|docker)" | head -20

Repository: openshift/external-secrets-operator

Length of output: 214

🌐 Web query:

OpenShift 4.21 early candidate ec.3 tag stability registry.ci.openshift.org

💡 Result:

Short answer: The 4.21 candidate tag ec.3 is present as an early-candidate payload on the CI registries, but it is a CI/preview (not GA) payload and can be changed or replaced — treat it as unstable for production. Evidence: the release dashboards show 4.21.0-ec.3 accepted/listed, and the candidate-4.21 channel listing shows candidate-4.21 -> 4.21.0-ec.3 (last modified Nov 28, 2025). [1][2][3]

Sources:

  • OpenShift release dashboard (4.21 ec.* listings). [1]
  • candidate-4.21 channel listing showing 4.21.0-ec.3. [2]
  • multi-release dashboard / registry CI listing for 4.21 ec.* payloads. [3]

🌐 Web query:

OpenShift CI pipeline base image versioning ec early candidate tags

💡 Result:

Short answer: In OpenShift CI and release naming, "ec" in tags means an Early‑Candidate / Early‑Access pre‑release build. You’ll see tags like 4.21.0-ec.0 or component tags such as 4.12.0-ec.3 used for testing and preview releases (not final GA). Those builds appear in CI/release mirrors and in component lists and rpm-ostree/base-image version strings; they should be treated as pre-release/test images and referenced by full tag or digest when pinning in pipelines. [1][2][3]

Sources:

  • Example release tag 4.21.0-ec.0 (microshift GitHub releases). [1]
  • Red Hat blog/console output showing components with 4.12.0-ec.3. [2]
  • OpenShift issue discussion showing use of "ec" candidate images in CI flows. [3]

Address early-candidate image instability risk in CI pipeline.

The base image tag 4.21.0-ec.3 is an Early-Candidate (pre-release) build, not a GA release. EC tags are explicitly marked as CI/preview payloads and can be changed, replaced, or removed. Using pre-release images for CI pipelines introduces instability risk. Verify whether this is intentional for testing purposes or if a stable release tag should be used instead. If early-candidate images are required, ensure the pipeline tolerates potential changes to the image or has fallback mechanisms.

🤖 Prompt for AI Agents 
images/ci/Dockerfile lines 17-17: the Dockerfile uses an Early-Candidate base
image tag (4.21.0-ec.3) which is unstable for CI; either replace it with a
confirmed GA/stable release tag (or preferably pin to a specific stable image
digest) or, if the EC image is intentional for testing, document that choice in
the repo and add robustness: implement a fallback to a known-good image (or
image digest) and/or make the CI tolerate image changes by parameterizing the
base image via a pipeline variable so it can be quickly switched; update the
Dockerfile reference and CI pipeline variables/documentation accordingly.

ARG SRC_DIR=/go/src/github.com/openshift/external-secrets-operator
COPY --from=builder $SRC_DIR/external-secrets-operator /bin/external-secrets-operator
USER 65534:65534
Expand Down
3 changes: 1 addition & 2 deletions images/ci/operand.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,7 @@ RUN git clone --depth 1 --branch $RELEASE_BRANCH https://github.com/openshift/ex
RUN go mod vendor
RUN go build -mod=vendor -tags $GO_BUILD_TAGS -o _output/external-secrets main.go

FROM registry.access.redhat.com/ubi9-minimal:9.4

FROM registry.ci.openshift.org/rhcos-devel/ocp-4.21-10.1:4.21.0-ec.3-x86_64
Copy link

@coderabbitai coderabbitai bot Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

# First, let's look at the operand.Dockerfile file to see the full context
cat -n images/ci/operand.Dockerfile

Repository: openshift/external-secrets-operator

Length of output: 214

🏁 Script executed:

# Also check the referenced images/ci/Dockerfile for comparison
cat -n images/ci/Dockerfile

Repository: openshift/external-secrets-operator

Length of output: 214

🏁 Script executed:

# Search for any other uses of these base images in the repo
rg "registry.ci.openshift.org" --type dockerfile -B 1 -A 1

Repository: openshift/external-secrets-operator

Length of output: 214

🌐 Web query:

RHEL 9 to RHEL 10 binary compatibility CGO compiled binaries

💡 Result:

Short answer: cgo-built Go binaries are not guaranteed to be 100% ABI‑stable across RHEL major releases because they link to the system C runtime and other native libraries. In practice:

  • A binary compiled on RHEL 9 will usually run on RHEL 10 if it only uses symbols/APIs present in RHEL 9’s libraries, because newer glibc versions normally remain backward‑compatible for existing symbols. [1][2]
  • RHEL 10 introduces new toolchain and library versions (glibc 2.39, GCC 14.x, zlib‑ng, different TBB, changes to long double ABI, etc.) and some libraries/packages are changed or replaced; those changes can break cgo binaries that depend on specific library ABIs or removed packages. Rebuilds may be required for components that changed ABI. [1][3][5]
  • Red Hat’s compatibility guidance classifies ABIs and explicitly documents compatibility windows and when you must rebuild; consult the RHEL 10 Application Compatibility/Adoption notes for specifics. [2][3]

Practical recommendations

  • If you need the widest portability, build on the oldest supported platform you intend to run on (build on RHEL 9 / UBI9 to run on RHEL 9 and usually RHEL 10). [2][6]
  • Where possible avoid using non‑stable/undocumented C APIs, and prefer linking to stable interfaces.
  • Run functional and integration tests on RHEL 10 (and any changed libraries like zlib/tbb) before deploying.
  • If you must guarantee runtime independence, consider fully static linking or distributing a compatible runtime (note: static glibc has limitations and is not always practical).
  • If you see breakage tied to a specific library (zlib‑ng, TBB, long double ABI, removed packages), rebuild the cgo code on RHEL 10 or vendor the needed library. [1][3][5]

Sources

  • RHEL 10 “Considerations in adopting RHEL 10” (lists glibc/GCC versions and incompatible package changes). [1]
  • Red Hat “RHEL 10: Application Compatibility / ABI compatibility” guidance. [2]
  • RHEL 10 package/change notes (zlib‑ng, TBB, long double ABI notes). [3]
  • RHEL container compatibility matrix (practical host/image compatibility guidance). [6]

If you want, tell me (a) whether your cgo binary is built static or dynamically, and (b) which libraries/symbols it depends on (ldd / objdump -T), and I’ll check likely compatibility and suggest next steps.

🌐 Web query:

strictfipsruntime OpenSSL RHEL 9 RHEL 10 compatibility

💡 Result:

Short answer

  • strictfipsruntime is a Go toolchain/GOEXPERIMENT option (from the golang‑fips downstream) that enables “strict FIPS runtime” checks: a binary built with it will panic at startup if it detects it’s running in a FIPS environment but cannot find/use a compatible OpenSSL/FIPS provider. [1][2]

RHEL 9 vs RHEL 10 compatibility considerations

  • RHEL 9 and RHEL 10 ship different OpenSSL/provider packaging and crypto‑policy defaults (both use OpenSSL 3.x in modern releases, but RHEL 10 removes compat-openssl11 and tightens default policies). That means a strictfipsruntime-built Go binary can run on RHEL 9 or 10 only if the target host provides a compatible libcrypto/OpenSSL + FIPS provider and acceptable crypto policy. If the binary cannot dlopen the expected OpenSSL/FIPS provider on that OS, it will crash under strictfipsruntime. [2][3]

Practical guidance

  • Test your binaries on the exact RHEL minor you will run (9.x vs 10.x). Confirm presence of a compatible libcrypto.so (OpenSSL 3 FIPS provider) and that /proc/sys/crypto/fips_enabled (or OPENSSL_FORCE_FIPS_MODE/GOLANG_FIPS env vars) reflect the FIPS state you intend. Use GO_OPENSSL_VERSION_OVERRIDE for forced testing if needed. If you must support older ABI/compat libraries, note RHEL 10 removed compat-openssl11. [1][2][3]

Sources

  1. golang-fips/go (strictfipsruntime, runtime checks, GOEXPERIMENT). [GitHub / pkg docs].
  2. Red Hat Developer: FIPS mode for Red Hat Go Toolset (behavior, OpenSSL selection, env vars).
  3. Red Hat docs — RHEL 10 security / considerations (compat-openssl11 removal, crypto‑policy changes, EMS/TLS differences).

🌐 Web query:

OpenShift RHCOS early candidate ec image stability production

💡 Result:

Short answer: Early Candidate (EC / candidate) RHCOS images are pre-release/testing builds and are not intended for production use — use GA images from the fast/stable channels for production clusters. See Red Hat’s release-channel guidance (candidate = release-candidate/testing; fast/stable = production-supported) and Red Hat examples that use EC images for testing/labs only. [1][2]

References

  • OpenShift release-channel guidance (candidate vs fast/stable — candidate contains release candidates and is not supported for production). [1]
  • Red Hat blog demonstrating EC RHCOS usage in lab/early-access scenarios (shows EC used for testing). [2]

🌐 Web query:

builder stage runtime OS version mismatch best practices container

💡 Result:

Short guidance and practical best practices when your build (builder) stage OS/version differs from the runtime stage in a container image:

  1. Prefer true multi-stage builds and copy only artifacts into the final runtime image (don’t carry build tools). This reduces size and attack surface. [1][5]

  2. Pin exact base-image tags for both builder and runtime (no floating "latest") so builds are reproducible. [5]

  3. Match ABI/Libc/architecture expectations:

    • Know whether your build artifacts require glibc vs musl (Alpine uses musl; Debian/Ubuntu use glibc). A glibc-linked binary will often fail on musl. Either build for the runtime libc or pick a runtime base that provides the expected libc. [2][3]

  4. When possible produce portable binaries:

    • Use static linking (or build with CGO_ENABLED=0 for Go) or build against the same libc used by the runtime. Static or musl-built binaries avoid libc mismatches. [3][7]

  5. If you must run glibc-built binaries on musl images, only as a last resort use compatibility packages (gcompat/libc6-compat) or a glibc shim — but be aware of compatibility and security trade-offs. Prefer changing the runtime base instead. [3][2]

  6. Use minimal, well-audited runtime images (distroless/scratch/alpine/slim) but only after validating compatibility of native extensions and system libraries. [1][4]

  7. Test the final image in an environment that mirrors production (including OS version, libc, locales, timezone, seccomp/AppArmor policies) to catch runtime-only failures early. Automate smoke tests in CI. [1][4]

  8. Expose and control build-time vs runtime configuration:

    • Pass needed compile-time flags / build args.
    • Avoid embedding build-time secrets; use build-time secrets features or CI secret handling. [5]

  9. Keep runtime surface minimal:

    • Copy only required libraries and files; remove temp/build artefacts. Scan final image for known vulnerabilities before deployment. [5][4]

  10. Document and enforce policies:

    • Specify supported base images, libc expectations, and how to produce compatible artifacts in your README/CI. Version-control Dockerfiles and base-image pins to make rollbacks straightforward. [5][1]

If you want, tell me the language/toolchain (Go, Java, Python, Node, C/C++) and the builder/runtime images you currently use and I’ll give concrete Dockerfile patterns and exact flags to avoid ABI/runtime mismatches.

Update to stable RHCOS GA image and verify runtime compatibility before deployment.

The operand binary is built with CGO_ENABLED=1 and strictfipsruntime, making it sensitive to RHEL version differences. Two concrete issues:

  1. Early candidate image not suitable for production: The -ec.3 tag indicates an early/pre-release RHCOS image, not a supported GA release. Use a stable GA image from the fast or stable release channels instead.

  2. strictfipsruntime requires RHEL 10 verification: RHEL 10 removed compat-openssl11 and changed OpenSSL/crypto-policy defaults. The strictfipsruntime runtime check will fail if the FIPS provider or compatible OpenSSL is unavailable. Test the compiled operand binary on the exact RHEL 10 image before deployment to confirm libcrypto.so and the FIPS provider are present and accessible.

Consider aligning the builder stage to RHEL 10 to match the final runtime and follow multi-stage build best practices.

🤖 Prompt for AI Agents 
In images/ci/operand.Dockerfile around line 15, the base image uses an early
candidate RHCOS (-ec.3) which is not a GA release and may be incompatible with a
CGO-enabled binary using strictfipsruntime; replace the FROM with a stable GA
RHCOS image from the fast or stable release channel (pick the appropriate 4.21
GA tag), align your builder stage to the same RHEL/OS major (RHEL 10) used at
runtime to avoid ABI/openssl mismatches, and before deployment run the compiled
operand binary inside the chosen RHEL10/RHCOS GA image to verify libcrypto.so
and the FIPS provider are present and accessible (adjust packaging or install
compat-openssl11/polyfill equivalents if needed) so strictfipsruntime checks
succeed.

ARG SRC_DIR=/go/src/github.com/openshift/external-secrets
COPY --from=builder $SRC_DIR/_output/external-secrets /bin/external-secrets

Expand Down