Skip to content

Commit ed6a85b

Browse files
xtremekforeverxtremekforever
authored
Support Debian 11 and Debian 12 distributions in the generator (#203)
This completes #116 and can be used to generate Debian Swift SDKs either by downloading dependencies from mirrors directly or from containers. - Added Debian `bullseye` and `bookworm` to `LinuxDistribution` and modded `SwiftSDKGenerator+Download` to work downloading both Ubuntu and Debian packages from their respective mirrors. - Swift SDKs can be generated from Docker using the `swift:*-bookworm` container for Debian 12, but for Debian 11 you must create the container yourself/manually put the Swift toolchain into it. - Added special handling in `VersionsConfiguration` so that if Debian 11 (bullseye) is selected, the Ubuntu 20.04 toolchain is downloaded. If Debian 12 (bookworm) is selected for Swift 5.9.* or Swift 5.10, the Ubuntu 22.04 toolchain is used instead. https://askubuntu.com/a/445496 - Added tests for the special handling in `LinuxRecipeTests.testItemsToDownloadForDebianTargets()` to ensure the correct toolchain is selected. To use: ```bash $ swift run swift-sdk-generator make-linux-sdk --linux-distribution-name debian --linux-distribution-version 11 $ swift run swift-sdk-generator make-linux-sdk --linux-distribution-name debian --linux-distribution-version 12 $ swift run swift-sdk-generator make-linux-sdk --linux-distribution-name debian --linux-distribution-version 12 --with-docker ``` I also added EndToEndTests for all Debian supported combinations, resulting in another 30GB of generated Swift SDKs: ``` 2.1 GiB [####################] /debian_12_x86_64_6.0.3-RELEASE_with-docker.artifactbundle 2.1 GiB [################### ] /debian_12_aarch64_6.0.3-RELEASE_with-docker.artifactbundle 2.0 GiB [################## ] /debian_12_x86_64_5.10.1-RELEASE_with-docker.artifactbundle 1.9 GiB [################# ] /debian_12_x86_64_6.0.3-RELEASE.artifactbundle 1.9 GiB [################# ] /debian_12_aarch64_6.0.3-RELEASE.artifactbundle 1.9 GiB [################# ] /debian_11_x86_64_6.0.3-RELEASE.artifactbundle 1.9 GiB [################# ] /debian_12_aarch64_5.10.1-RELEASE_with-docker.artifactbundle 1.9 GiB [################# ] /debian_11_aarch64_6.0.3-RELEASE.artifactbundle 1.8 GiB [################ ] /debian_12_x86_64_5.10.1-RELEASE.artifactbundle 1.7 GiB [################ ] /debian_12_aarch64_5.10.1-RELEASE.artifactbundle 1.7 GiB [################ ] /debian_11_x86_64_5.10.1-RELEASE.artifactbundle 1.7 GiB [############### ] /debian_11_aarch64_5.10.1-RELEASE.artifactbundle 1.7 GiB [############### ] /debian_12_x86_64_5.9.2-RELEASE.artifactbundle 1.7 GiB [############### ] /debian_12_aarch64_5.9.2-RELEASE.artifactbundle 1.7 GiB [############### ] /debian_11_x86_64_5.9.2-RELEASE.artifactbundle 1.6 GiB [############### ] /debian_11_aarch64_5.9.2-RELEASE.artifactbundle ``` To make this work properly, I modded the `targetSwift` path in `DownloadableArtifacts` to use the swiftPlatform name in the file name to avoid the issues I saw with the EndToEndTests, where the *.tar.gz files would get corrupted when trying to download a different version of the target Swift on top of the existing file. Now, they look like this: ``` target_swift_5.10.1-RELEASE_debian12_aarch64.tar.gz target_swift_5.9.2-RELEASE_ubuntu22.04_aarch64.tar.gz target_swift_5.10.1-RELEASE_debian12_x86_64.tar.gz target_swift_5.9.2-RELEASE_ubuntu22.04_x86_64.tar.gz target_swift_5.10.1-RELEASE_ubuntu20.04_aarch64.tar.gz target_swift_6.0.3-RELEASE_debian12_aarch64.tar.gz target_swift_5.10.1-RELEASE_ubuntu20.04_x86_64.tar.gz target_swift_6.0.3-RELEASE_debian12_x86_64.tar.gz target_swift_5.10-RELEASE_ubuntu20.04_aarch64.tar.gz target_swift_6.0.3-RELEASE_ubuntu20.04_aarch64.tar.gz target_swift_5.9.2-RELEASE_ubuntu20.04_aarch64.tar.gz target_swift_6.0.3-RELEASE_ubuntu20.04_x86_64.tar.gz target_swift_5.9.2-RELEASE_ubuntu20.04_x86_64.tar.gz ```
1 parent 9fe010c commit ed6a85b

File tree

11 files changed

+615
-156
lines changed

11 files changed

+615
-156
lines changed

‎README.md

+3-3
Original file line numberDiff line numberDiff line change
@@ -47,14 +47,14 @@ The generator also allows cross-compiling between any Linux distributions offici
4747
| macOS (arm64) | ✅ macOS 13.0+ | ❌ |
4848
| macOS (x86_64) | ✅ macOS 13.0+[^1] | ❌ |
4949
| Ubuntu | ✅ 20.04+ | ✅ 20.04+ |
50-
| RHEL | ✅ Fedora 39[^2], UBI 9 | ✅ UBI 9 |
50+
| Debian | ✅ 11, 12[^2] | ✅ 11, 12[^2] |
51+
| RHEL | ✅ Fedora 39, UBI 9 | ✅ Fedora 39, UBI 9[^3] |
5152
| Amazon Linux 2 | ✅ Supported | ✅ Supported[^3] |
52-
| Debian 12 | ✅ Supported[^2] | ✅ Supported[^2][^3] |
5353

5454
[^1]: Since LLVM project doesn't provide pre-built binaries of `lld` for macOS on x86_64, it will be automatically built
5555
from sources by the generator, which will increase its run by at least 15 minutes on recent hardware. You will also
5656
need CMake and Ninja preinstalled (e.g. via `brew install cmake ninja`).
57-
[^2]: These distributions are only supported by Swift 5.10.1 and later as both host and target platforms.
57+
[^2]: Swift does not officially support Debian 11 or Debian 12 with Swift versions before 5.10.1. However, the Ubuntu 20.04/22.04 toolchains can be used with Debian 11 and 12 (respectively) since they are binary compatible.
5858
[^3]: These versions are technically supported but require custom commands and a Docker container to build the Swift SDK, as the generator will not download dependencies for these distributions automatically. See [issue #138](https://github.com/swiftlang/swift-sdk-generator/issues/138).
5959

6060
## How to use it

‎Sources/GeneratorCLI/GeneratorCLI.swift

+7-3
Original file line numberDiff line numberDiff line change
@@ -199,7 +199,8 @@ extension GeneratorCLI {
199199

200200
@Option(
201201
help: """
202-
Linux distribution to use if the target platform is Linux. Available options: `ubuntu`, `rhel`. Default is `ubuntu`.
202+
Linux distribution to use if the target platform is Linux.
203+
- Available options: `ubuntu`, `debian`, `rhel`. Default is `ubuntu`.
203204
""",
204205
transform: LinuxDistribution.Name.init(nameString:)
205206
)
@@ -208,8 +209,9 @@ extension GeneratorCLI {
208209
@Option(
209210
help: """
210211
Version of the Linux distribution used as a target platform.
211-
Available options for Ubuntu: `20.04`, `22.04` (default when `--linux-distribution-name` is `ubuntu`), `24.04`.
212-
Available options for RHEL: `ubi9` (default when `--linux-distribution-name` is `rhel`).
212+
- Available options for Ubuntu: `20.04`, `22.04` (default when `--distribution-name` is `ubuntu`), `24.04`.
213+
- Available options for Debian: `11`, `12` (default when `--distribution-name` is `debian`).
214+
- Available options for RHEL: `ubi9` (default when `--distribution-name` is `rhel`).
213215
"""
214216
)
215217
var linuxDistributionVersion: String?
@@ -239,6 +241,8 @@ extension GeneratorCLI {
239241
linuxDistributionDefaultVersion = "ubi9"
240242
case .ubuntu:
241243
linuxDistributionDefaultVersion = "22.04"
244+
case .debian:
245+
linuxDistributionDefaultVersion = "12"
242246
}
243247
let linuxDistributionVersion =
244248
self.linuxDistributionVersion ?? linuxDistributionDefaultVersion

‎Sources/SwiftSDKGenerator/Generator/SwiftSDKGenerator+Copy.swift

+32-5
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ extension SwiftSDKGenerator {
2222
logger.info("Launching a container to extract the Swift SDK for the target triple...")
2323
try await withDockerContainer(fromImage: baseDockerImage) { containerID in
2424
try await inTemporaryDirectory { generator, _ in
25+
let sdkLibPath = sdkDirPath.appending("lib")
2526
let sdkUsrPath = sdkDirPath.appending("usr")
2627
try await generator.createDirectoryIfNeeded(at: sdkUsrPath)
2728
try await generator.copyFromDockerContainer(
@@ -60,6 +61,15 @@ extension SwiftSDKGenerator {
6061
to: sdkUsrLib64Path
6162
)
6263
try await createSymlink(at: sdkDirPath.appending("lib64"), pointingTo: "./usr/lib64")
64+
} else if case let containerLib64 = FilePath("/lib64"),
65+
try await generator.doesPathExist(containerLib64, inContainer: containerID)
66+
{
67+
let sdkLib64Path = sdkDirPath.appending("lib64")
68+
try await generator.copyFromDockerContainer(
69+
id: containerID,
70+
from: containerLib64,
71+
to: sdkLib64Path
72+
)
6373
}
6474

6575
let sdkUsrLibPath = sdkUsrPath.appending("lib")
@@ -72,13 +82,26 @@ extension SwiftSDKGenerator {
7282
// architecture-specific directories:
7383
// https://wiki.ubuntu.com/MultiarchSpec
7484
// But not in all containers, so don't fail if it does not exist.
75-
if case .ubuntu = targetDistribution {
76-
subpaths += [("\(targetTriple.archName)-linux-gnu", false)]
85+
if targetDistribution.name == .ubuntu || targetDistribution.name == .debian {
86+
var archSubpath = "\(targetTriple.archName)-linux-gnu"
7787

78-
// Custom subpath for armv7
88+
// armv7 with Debian uses a custom subpath for armhf
7989
if targetTriple.archName == "armv7" {
80-
subpaths += [("arm-linux-gnueabihf", false)]
90+
archSubpath = "arm-linux-gnueabihf"
8191
}
92+
93+
// Copy /lib/<archSubpath> for Debian 11
94+
if case let .debian(debian) = targetDistribution, debian == .bullseye {
95+
try await generator.createDirectoryIfNeeded(at: sdkLibPath)
96+
try await generator.copyFromDockerContainer(
97+
id: containerID,
98+
from: FilePath("/lib").appending(archSubpath),
99+
to: sdkLibPath.appending(archSubpath),
100+
failIfNotExists: false
101+
)
102+
}
103+
104+
subpaths += [(archSubpath, false)]
82105
}
83106

84107
for (subpath, failIfNotExists) in subpaths {
@@ -89,7 +112,11 @@ extension SwiftSDKGenerator {
89112
failIfNotExists: failIfNotExists
90113
)
91114
}
92-
try await generator.createSymlink(at: sdkDirPath.appending("lib"), pointingTo: "usr/lib")
115+
116+
// Symlink if we do not have a /lib directory in the SDK
117+
if await !generator.doesFileExist(at: sdkLibPath) {
118+
try await generator.createSymlink(at: sdkLibPath, pointingTo: "usr/lib")
119+
}
93120

94121
// Look for 32-bit libraries to remove from RHEL-based distros
95122
// These are not needed, and the amazonlinux2 x86_64 symlinks are messed up

0 commit comments

Comments
 (0)