Skip to content

Conversation

mergify[bot]
Copy link
Contributor

@mergify mergify bot commented Sep 5, 2025

What is the problem this PR solves?

The fleet-server binary was previously built directly on VM runners instead of using a controlled environment such as golang-crossbuild.
This caused the binary to be dynamically linked against the host system’s GLIBC, introducing higher GLIBC requirements (≥ 2.32) than those declared in the Elastic OS support matrix (≥ 2.17).

Note that, even if the build configuration used CGO_ENABLED=0 (which didn't before this PR) together with -buildmode=pie, on Linux, this combination still produces a dynamically linked binary because pie relies on the dynamic loader (e.g. /lib/ld-linux-aarch64.so.1 on arm64).

How does this PR solve the problem?

  • Switches builds for all packaging steps to run inside golang-crossbuild images (via mage docker:release), avoiding linkage against host GLIBC.
  • Forces CGO_ENABLED=0 across builds (except for FIPS variants, which require CGO).
  • Removes -buildmode=pie so that binaries are fully static, with no interpreter requirement.
  • Updates Buildkite pipelines to:
    • Add explicit packaging steps for x86_64 and arm64 (including FIPS variants).
    • Save packaging artifacts (build/distributions/**) for DRA validation and debugging.
  • Fixes an issue with mage docker:image when compiling FIPS variants.

As a result, Fleet Server binaries are now fully static and do not have a GLIBC requirement, restoring compatibility with Amazon Linux 2 and other supported distributions.

For example, you can download the Linux packaging artifacts from the CI steps introduced in this PR:

Running file against them after extraction shows static binaries:

$ file ~/Downloads/fleet-server-9.2.0-linux-arm64/fleet-server
/Users/pkoutsovasilis/Downloads/fleet-server-9.2.0-linux-arm64/fleet-server: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=22095f2a8f3da7f6271dd858dcc13fe86e19acf2, stripped

$ file ~/Downloads/fleet-server-9.2.0-linux-x86_64/fleet-server
/Users/pkoutsovasilis/Downloads/fleet-server-9.2.0-linux-x86_64/fleet-server: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=f9a203f5c78161f88d9e10be1bf5863f970adcac, stripped

How to test this PR locally

mage docker:binary
docker run --rm -it --entrypoint /app/fleet-server -v $PWD/build/binaries/fleet-server-9.2.0-linux-arm64:/app amazonlinux:2 --version

Design Checklist

  • I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
  • I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
  • I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in ./changelog/fragments using the changelog tool

Related issues


This is an automatic backport of pull request #5392 done by [Mergify](https://mergify.com).

* fix: package fleet-server using golang-crossbuild and introduce packaging steps on every PR

* feat: save artifacts during packaging for DRA

* fix: disable CGO and buildmode pie

* doc: add changelog fragment

(cherry picked from commit 7da3f69)

# Conflicts:
#	.buildkite/pipeline.package.mbp.yml
#	.buildkite/pipeline.yml
#	.buildkite/scripts/package.sh
#	Dockerfile.fips
#	magefile.go
@mergify mergify bot requested a review from a team as a code owner September 5, 2025 08:25
@mergify mergify bot added backport conflicts There is a conflict in the backported pull request labels Sep 5, 2025
@mergify mergify bot requested review from michel-laterman and pchila September 5, 2025 08:25
Copy link
Contributor Author

mergify bot commented Sep 5, 2025

Cherry-pick of 7da3f69 has failed:

On branch mergify/bp/8.18/pr-5392
Your branch is up to date with 'origin/8.18'.

You are currently cherry-picking commit 7da3f69.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1756940128-fix-binary-compilation.yaml

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	both modified:   .buildkite/pipeline.package.mbp.yml
	both modified:   .buildkite/pipeline.yml
	both modified:   .buildkite/scripts/package.sh
	both modified:   Dockerfile.fips
	deleted by us:   magefile.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

@prodsecmachine
Copy link

prodsecmachine commented Sep 5, 2025

🎉 Snyk checks have passed. No issues have been found so far.

security/snyk check is complete. No issues have been found. (View Details)

license/snyk check is complete. No issues have been found. (View Details)

@github-actions github-actions bot added bug Something isn't working Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team labels Sep 5, 2025
Copy link
Contributor Author

mergify bot commented Sep 8, 2025

This pull request has not been merged yet. Could you please review and merge it @pkoutsovasilis? 🙏

Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

@pkoutsovasilis pkoutsovasilis merged commit 1a5ec7f into 8.18 Sep 8, 2025
8 checks passed
@pkoutsovasilis pkoutsovasilis deleted the mergify/bp/8.18/pr-5392 branch September 8, 2025 10:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport bug Something isn't working conflicts There is a conflict in the backported pull request Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants