Simplify Swift SDK installation/selection with aliases

[MaxDesiatov]

Motivation:

As Swift SDKs require host toolchain version to exactly match the version used to build a Swift SDK that's used, we need to automate Swift SDK installation and selection process. Proposed solution is for the users to rely on predefined aliases, that SwiftPM uses as inputs to compute the exact Swift SDK installation URL and selection ID. New CLI options are added, which in the future could be unified with existing CLI arguments and options for Swift SDKs.

Modifications:

Implemented for installation of Swift SDKs from swift.org:

swift sdk install --experimental-alias static-linux

For swift build, swift run, and swift test:

swift build --experimental-swift-sdk-alias static-linux

This assumes version.json is available in usr/lib/swift directory of the host toolchain, looking like this

{
  "branch": "development",
  "tag": "DEVELOPMENT-SNAPSHOT-2025-04-24-a",
  "platform": "ubuntu2004",
  "architecture": "aarch64"
}

Information from this file is used to compute Swift SDK URLs and IDs.

Error messages are shown when either version.json is not found or an unknown alias is used.

Result:

Users no longer need to find the exactly matching Swift SDK for the currently selected toolchain version.

Resolves rdar://150227998.

[MaxDesiatov]

@swift-ci test

[jakepetroules]

Is there any way this could be more data driven, so we don't have to change the toolchain every time there's a new Swift SDK? Or is this meant to be geared towards supporting just the Swift SDKs that are officially distributed by swift.org, and you need to do manual selection for others?

[jakepetroules]

This should be changed to a real message before merging.

[kateinoigakukun]

Totally agree with @jakepetroules. I think we should explore more data-driven way for installation process by even changing the current distribution scheme. At least, SwiftPM should not know about the list of aliases.

[MaxDesiatov]

Is there any way this could be more data driven, so we don't have to change the toolchain every time there's a new Swift SDK?

@jakepetroules Can you elaborate on what you mean by "data driven"? As for changing the toolchain, we don't build Swift SDKs for older toolchains, so new Swift SDKs would require new toolchains either way.

Or is this meant to be geared towards supporting just the Swift SDKs that are officially distributed by swift.org, and you need to do manual selection for others?

Yes

[MaxDesiatov]

Totally agree with @jakepetroules. I think we should explore more data-driven way for installation process by even changing the current distribution scheme. At least, SwiftPM should not know about the list of aliases.

@kateinoigakukun Can you elaborate? How would it get the list of aliases otherwise? Any server-based solutions are out of scope for this change.

We can only rely on changes to client-side tools distributed to users at this moment. Relying on external sources is not possible due to multiple reasons.

[al45tair]

As Swift SDKs require host toolchain version to exactly match the version used to build a Swift SDK that's used

I keep wondering whether we should really be fixing that problem, instead of trying to work around it (either with this PR or by giving people complicated instructions about matching them up). The fundamental issue here is that we aren't ABI stable, so we're relying on swiftmodule files, which are themselves not ABI stable and that's why the toolchain and SDK must match.

Totally agree with @jakepetroules. I think we should explore more data-driven way for installation process by even changing the current distribution scheme. At least, SwiftPM should not know about the list of aliases.

I think I also agree with that sentiment. Could we perhaps have an option to specify the SDK by pointing at a JSON file that provides the required information to locate the actual SDK according to the current toolchain version? So, e.g. you'd write something like

$ swift sdk install swift.org/sdks/static-linux

or (to give a purely imaginary third-party example)

$ swift sdk install microsoft.com/swift-sdks/xbox

Then if the JSON data at the given URL doesn't contain a version matching the toolchain, we'd respond with something like

Unable to find a matching SDK for this toolchain (<toolchain version>). Available versions are:

<list of available SDK versions>

[MaxDesiatov]

I keep wondering whether we should really be fixing that problem, instead of trying to work around it (either with this PR or by giving people complicated instructions about matching them up). The fundamental issue here is that we aren't ABI stable, so we're relying on swiftmodule files, which are themselves not ABI stable and that's why the toolchain and SDK must match.

Making Swift ABI-stable on non-Darwin platforms is out of scope of this PR 🙂

I think I also agree with that sentiment. Could we perhaps have an option to specify the SDK by pointing at a JSON file that provides the required information to locate the actual SDK according to the current toolchain version?

@al45tair That's the problem, what would generate that JSON? As server-side solutions are out of scope (due to time constraints), we'd have to hardcode that JSON somewhere, either in SwiftPM or in the toolchain. Since SwiftPM is responsible for Swift SDK installation, hardcoding it in SwiftPM makes the most sense. And as we're already hardcoding it, JSON brings no benefits when compared to just writing this logic in plain Swift.

[MaxDesiatov]

@swift-ci test self hosted windows

[al45tair]

@al45tair That's the problem, what would generate that JSON? As server-side solutions are out of scope (due to time constraints), we'd have to hardcode that JSON somewhere, either in SwiftPM or in the toolchain. Since SwiftPM is responsible for Swift SDK installation, hardcoding it in SwiftPM makes the most sense. And as we're already hardcoding it, JSON brings no benefits when compared to just writing this logic in plain Swift.

We'd hand-write it (initially — though it might in the longer term be automatically generated by the CI jobs) and put it on the web site (as opposed to in the source code for SwiftPM). So e.g. if you wrote

$ swift sdk install swift.org/sdks/static-linux

that would try to fetch the JSON file at e.g. https://swift.org/sdks/static-linux/sdk-index.json.

[MaxDesiatov]

that would try to fetch the JSON file at e.g. https://swift.org/sdks/static-linux/sdk-index.json.

As mentioned above, any server-side solutions are out of scope of this PR. We're looking for something very targeted and focused that can resolve the installation issues for user's in the short term. When a solution for hosting a Swift SDKs index on swift.org is available, I'd be very happy to implement it in a separate PR.

[kateinoigakukun]

As mentioned above, any server-side solutions are out of scope of this PR.

What do you mean by "server-side solutions"? I think we can implement something what @al45tair suggested by putting static files like what we do in https://github.com/swiftwasm/swift-sdk-index or https://github.com/swiftlang/swift-org-website/blob/main/api/v1/install/dev/6.0/static-sdk.json, and it shouldn't require "server" process other than just a static HTTP server we already have on swift.org.

I'm ok with hardcoding the SDK source URL locations and limiting the alias with known ones for now for time constraints, but I think we should at least define what it will be eventually to allow third-parties to provide their SDK index and we need to make sure we can implement it without changing CLI interface.

[MaxDesiatov]

What do you mean by "server-side solutions"? I think we can implement something what @al45tair suggested by putting static files like what we do in https://github.com/swiftwasm/swift-sdk-index or https://github.com/swiftlang/swift-org-website/blob/main/api/v1/install/dev/6.0/static-sdk.json, and it shouldn't require "server" process other than just a static HTTP server we already have on swift.org.

I mean deploying payloads on any server, including a static server. This suggestion glosses over complexities of deployment on an arbitrary static server. What's very easy to set up in a GitHub organization with GitHub Actions can't be universally replicated in all scenarios. There's a lot of mechanics involved in keeping these payloads up-to-date, tying them to existing Swift CI jobs etc. That's definitely out of scope of this PR. When these deployment and CI issues are resolved, I'd very happy to come back and overhaul everything to be dynamically configurable, generalized, and clean, avoiding any hardcoding etc. But before that we need something that solves immediate problems that users already have been having for quite a long time.

[MaxDesiatov]

@swift-ci test self hosted windows

[MaxDesiatov]

Is there any way this could be more data driven, so we don't have to change the toolchain every time there's a new Swift SDK? Or is this meant to be geared towards supporting just the Swift SDKs that are officially distributed by swift.org, and you need to do manual selection for others?

@jakepetroules do you have any remaining objections here? This PR is purely for simplifying interactions with few existing Swift SDKs from swift.org. To summarize the conversion on this thread: I'd be super happy to make this configurable with some manifest hosted on swift.org website, but that has to be implemented by swift.org infrastructure team first and is out of scope of this PR.

I still think hardcoding URL components for now is a significant improvement, while configurability remains an implementation detail that we can add later when necessary server-side components are ready. And to be extra-safe, this has an --experimental prefix to give users an opportunity to try it out and for us to get feedback from them as soon as possible.