Rewrite release CI

[hasufell]

This is a rewrite and an extension of #9437

The main motivation is to:

1. have a reliable release CI that runs (mostly) on unmanaged runners
2. have access to GitHubs CI infrastructure for faster and more reliable execution
3. unify the CI so that we only need to maintain one place/flavor of CI (ease of maintenance, less knowledge required for contributors etc.)

The major difference to the previous PR is that we run the "integration tests" of cabal-install (because that is the main part that diverges from the validate pipeline in terms of configuration and build flags).

[mpickering]

Perhaps it would be good to first establish what the goals with this PR is as it seems to be doing a few different things.

I think as a first step, migrating to github CI should match the configurations and artifacts produced by the existing release CI. Afterwards, once that is achieved then further platforms and packaging tweaks can be explored in a more controlled manner.

It seems risky to both migrate to a different CI platform, modify the supported platforms and also modify packaging options all in one commit. (Not that I am expressing a judgement about the value of also achieving those things as individual goals)

[hasufell]

I think as a first step, migrating to github CI should match the configurations and artifacts produced by the existing release CI

The gitlab configuration contains EOLed distros and lacks many of the artifacts that the current PR provides. Matching gitlab would be mostly a regression.

My impression is that the current release CI is not properly maintained, which is what this PR is trying to solve.

[mpickering]

Which EOL distros are you referring to exactly? It would be good to remove those certainly.

Adding new platforms for release configurations seems to be an additive task which can be achieved in follow-up discussions if cabal maintainers decide they wish to produce binaries for new platforms.

What additional artifacts are produced? How are those different from the existing release artifacts?

Just so we are clear here Julian, I agree with the premise of this patch. Perhaps if you consider the other changes as necessary/desirable as well, another way forward would be to produce a detailed list of what is the same/different to before, so each one can be considered individually during a review.

[hasufell]

Which EOL distros are you referring to exactly?

There are some updates I should do to this PR as well, but here is the inventory:

Tarball	GHCup	Cabal	Remark
x86_64-linux-deb10	-	+	EOL
x86_64-linux-deb11	+	+
x86_64-linux-deb12	+	+
x86_64-linux-fedora33	+	+	EOL
x86_64-linux-fedora36	-	+	EOL
x86_64-linux-fedora37	+	-	EOL
x86_64-linux-fedora38	-	+	EOL
x86_64-linux-rocky8	+	+
x86_64-linux-rocky9	+	-
x86_64-linux-ubuntu20_04	+	+
x86_64-linux-ubuntu22_04	+	+
x86_64-linux-ubuntu24_04	+	+
x86_64-linux-mint20_3	+	-	EOL
x86_64-linux-mint21_3	+	-
x86_64-linux-mint22_1	-	-
x86_64-linux-alpine3_12	+	+	EOL, static
x86_64-linux-alpine3_19	+	-	static
x86_64-linux-alpine3_20	-	+	static
x86_64-linux-void-glibc	+	-	rolling
x86_64-freebsd14_2	+	-
x86_64-darwin	+	+
x86_64-windows	+	+
aarch64-linux-deb10	-	+	EOL
aarch64-linux-deb11	+	+
aarch64-linux-deb12	-	+
aarch64-linux-alpine3_18	-	+	EOL, static
aarch64-linux-alpine3_21	+	-	static
aarch64-darwin	+	+
i386-linux-deb10	-	+	EOL
i386-linux-alpine3_12	-	+	EOL, static
i386-linux-alpine3_20	-	+	static
i386-linux-alpine3_21	+	-	static

[hasufell]

Our strategy going forward is as follows:

• we want to reduce the linux builds to 2:
  • one fully statically linked against musl
  • one dynamically linked against the oldest glibc that's not EOL, but statically against other libraries (gmp, ncurses etc)... this works via pointing --extra-lib-dirs to a dir which includes only static library archives (cabal only seems to link against gmp and I have a working example binary)
• then all known glibc based distros use the glibc one and alpine/unkown distros can use the static musl one

We end up with the following build environments:

Build env	Remark
x86_64-linux-rocky8	glibc
x86_64-linux-alpine3_22	musl
x86_64-freebsd14_2
x86_64-darwin
x86_64-windows
aarch64-linux-rocky8	glibc
aarch64-linux-alpine3_22	musl
aarch64-darwin
i386-linux-deb11	glibc
i386-linux-alpine3_22	musl

[mpickering]

Our strategy going forward is as follows:

* we want to reduce the linux builds to 2:
  
  * one fully statically linked against musl
  * one dynamically linked against the oldest glibc that's not EOL, but statically against other libraries (gmp, ncurses etc)... this works via pointing `--extra-lib-dirs` to a dir which includes only static library archives (cabal only seems to link against gmp and I have a working example binary)

* then all known glibc based distros use the glibc one and alpine/unkown distros can use the static musl one

This proposal seems to be a different approach to building artifacts than the cabal project has used in the past, could you open an issue to discuss this proposal with the other maintainers? It seems to be a different topic to this MR.

It would be good to build less binaries overall, perhaps this approach could be made easier by adding some features of options to cabal-install if it works out well.

[Mikolaj]

Our strategy going forward is as follows:

Indeed, your strategy seems very attractive and quite radical. Is it applicable to a different major project, such as GHC? or ghcup itself or something else major? I'd feel more comfortable if I could contant existing users of such a strategy and even more comfortable, if we could piggy-back on GHC, as we do currently, using CI very similar to what GHC uses and taking advantage of the professionalism of @chreekat and good will of the Haskell Foundation in the management of the release CI. Did GHC consider this strategy?

[hasufell]

if we could piggy-back on GHC, as we do currently, using CI very similar to what GHC uses and taking advantage of the professionalism of @chreekat and good will of the Haskell Foundation in the management of the release CI

@Mikolaj I find your response incredibly disrespectful as a contributor, because you're insinuating "we don't trust your expertise on this matter".

I'm being paid by IOG to do this type of work and we will do it anyway, with or without upstream. But as we've been asked by @bgamari and others during ZuriHac whether we intend to contribute back upstream, here is the answer: yes we do. And here is our proof. In turn, this is a great opportunity for the cabal team to demonstrate that they're willing to collaborate with other teams outside of their core "cabal".

Also as a side note: the Haskell Foundation itself apparently trusts my expertise on this matter as they are funding my proposal (and I am the person managing the Haskell Foundation GitHub runners).

That doesn't mean our goals align and I'm willing to make adjustments to this PR, but I can't work with you, unless the commentary remains professional and technical.

[Mikolaj]

Edited: I don't feel I can productively contribute to reviewing this PR, so let me thank @mpickering for discussing its merits up to this point and let ma ask @mpickering and other cabal maintainers and developers to carry on.

[mpickering]

Right, I separately chatted with Julian and Mikolaj. It does seem like there are some unresolved tensions from past interactions. Thank you for your gracious reflection Mikolaj that you would find engaging with this PR difficult.

With @hasufell I discussed the ways that we could move this PR forward, and that we should agree a shared plan of work.

Julian is quite keen to make the CI line-up more with what's required for ghcup, which is understandable since ghcup is the primary way which cabal binaries are distributed.

Cabal maintainers are keen to not have a system which relies on one person (Julian) to maintain, and wish to understand the new system carefully so they can use it without external help in future.

• Julian said to me what as a first step he could make the platforms in this PR align more, that seems like an uncontroversial way to make progress. But if maintaining platform compatibility is difficult for other reasons, we are not opposed to also updating distributed platforms at the same time.
• Matt can then take a careful look at the patch, with the mindful goal of something which can be maintained without Julian's assistance if necessary.
• The new approach to making binaries sounds interesting, but should be discussed on a ticket and via an experimental job before committing to go down that track for the actual releases. Julian will make a ticket where this can be discussed more broadly.

Does that match your impression from our conversation @hasufell ?

[hasufell]

@mpickering that sounds resonable.

Tomorrow I will start:

• splitting the PR into smaller chunks (e.g. the platform updates in a separate commit)
• moving the release stuff to a reusable workflow, so it can be triggered from other places
• open a ticket about reducing linux builds and how to achieve it

with the mindful goal of something which can be maintained without Julian's assistance if necessary.

Well, I think that the bus factor will be much better with the move to GitHub, since most cabal contributors already have exposure to GitHub CI and it doesn't require intimate knowledge of runners (except for FreeBSD, but I'm confident that @geekosaur also has the required expertise to handle it).

[ffaf1]

From CONTRIBUTING.md.

A pull request fixes a problem that is described in an issue. Make sure to file an issue before opening a pull request. In the issue you can illustrate your proposed design, UX considerations, tradeoffs etc. and work them out with other contributors. The PR itself is for implementation.

I would too like to ask for an issue with goals clearly stated for sure, and possibly trade-offs and resources considerations.

[hasufell]

@ffaf1 I can do that, but please note that the motivation and merits have been discussed for years and I would like to assume by now Cabal developers are well aware. There even have been synchronous calls about this topic and multiple Cabal developers/contributors have expressed positive interest in this approach.

So yes, I can open an issue with motivation and explanation, but I won't be participating in argumentative discussion, nor will I try to prove why GitHub is a better decision for release CI in the context of Cabal. This is up to you to decide.

This is work that we're using over at stable-haskell and I'm happy to upstream it and invest additional effort into making it upstreamable.

[hasufell]

#11078

[mpickering]

Thanks for making the ticket Julian, that's just a normal part of contributing to cabal. Having a place to discuss the issue which isn't on the PR is useful for the longevity of recording these discussions.

I'll get back to looking at this next week when I'm back at work if you have made the changes we discussed.

[hasufell]

Only thing left in this PR is executing on the "only 2 bindists for linux" plan in a separate commit.

[hasufell]

@mpickering thanks for your thorough review

[geekosaur]

TBH I feel like this should be in a separate PR. I can cut it and get it in if you'd prefer not to be juggling another one.

[hasufell]

If this is removed, then the PR CI fails, so I disagree that this needs to be in another PR.

[hasufell]

If you mean I need to make another PR for this, then wait until that's reviewed + merged and then continue with this PR... then I'm afraid I don't have the time for that. As I indicated above: I don't work on this in my free time and the time allocation for dealing with this PR is running out. It's been 2 weeks now.

[geekosaur]

I am still thinking it deserves its own PR and possibly issue for documentation reasons. Would you have a problem with my cherry-picking that commit into a separate PR without removing it here, so the two won't conflict?

[geekosaur]

(This is not uncommon with GHC PRs that depend on each other, BTW: use cherry-picking and let the first one to be merged win without conflicting with the other. I consider this a shortcoming of GitHub and GitLab and probably other similar UIs.)

[hasufell]

I am still thinking it deserves its own PR and possibly issue for documentation reasons. Would you have a problem with my cherry-picking that commit into a separate PR without removing it here, so the two won't conflict?

Sure

[geekosaur]

1. Didn't we previously nuke the lukko stuff, or at least mark it as default off?
2. You should probably update the comment to indicate that it only applies to -git-rev.

[hasufell]

It is not nuked, the flag is just off by default. I want to enforce that release is always built without it, due to bugs on 32bit. If someone messes with the cabal file flags, then the release may break. This makes sure that can't happen. The releases in general should never depend on default flags. All flags needed should be explicit.

[ulysses4ever]

Would it be possible to create a short README in the scripts/release folder with a sketch of a workflow with these new scripts? In particular, what order one is expected to call them in? This readme could be referenced from the release wiki page.

[hasufell]

Would it be possible to create a short README in the scripts/release folder with a sketch of a workflow with these new scripts? In particular, what order one is expected to call them in? This readme could be referenced from the release wiki page.

As @mpickering has indicated, this goes into the wiki: https://github.com/haskell/cabal/wiki/Making-a-release#c6-publishing-the-artifacts

I will update the wiki after this PR is merged, not before.

The entry point is download-gh-artifacts.sh and if you've read that script, you can see that it already gives further instructions on what to do:

https://github.com/haskell/cabal/pull/11072/files#diff-b3f6cc695dbf4e339fdc60df7ddd087b9efbec13d0cb15a5fc6d21b2ab97b7e5R31-R41

[ulysses4ever]

As @mpickering has indicated, this goes into the wiki: https://github.com/haskell/cabal/wiki/Making-a-release#c6-publishing-the-artifacts

I'll have to disagree with Matt on this one. That page is already bloated like hell and hard to navigate. We should strive to make it lighter, and this re-write is a great opportunity to do that. But we can agree to disagree.

The entry point is download-gh-artifacts.sh and if you've read that script, you can see that it already gives further instructions on what to do

I find a bunch of echo hard to read. It seems to favor the user who runs it as opposed to the user who read it, but we read these scripts MUCH more often than we run them, so I think it's the wrong direction. I'd prefer documentation in the form of comment at the top of the file, as it was the case before.

Because of the disagreements above, I don't feel confident to approve this PR, but I removed my "change-requested" block, so whenever Brandon is happy, it'll be easy to move on.

Thanks for all the effort you put in it! Remarkable work and hopefully the one that will make our release process smoother.

[hasufell]

@ulysses4ever sure... I want to avoid going back and forth with minor changes, especially when there's no clear consensus among cabal devs. I think the cabal team can figure that out after it's merged.

I'm happy to clear up specific questions though.

some of my concerns were addressed and some were pushed for "future work" around the Wiki