Problems with the planned new cross-compilation model

[finagolfin]

Description

@compnerd put together a document on a planned new cross-compilation model last summer, which replaces the flags -sdk and -resource-dir with the flags -sdk and -sysroot instead. I've since seen some issues with this new model, in order of importance:

1. Explicitly specifying an -sdk has a bunch of bugs and doesn't work very well. The most common way to ship Swift core modules and runtime libraries is next to the compiler instead and explicitly specifying an external -resource-dir works reasonably, though bugs have crept in there too. However, explicitly specifying an external -sdk, particularly a full SDK that contains both a C/C++ sysroot and a Swift resource directory, has historically been the least used configuration and comes with a lot of bugs.
   For example, I just saw that the Windows CI tries to build the Foundation macros for the Windows host by using a 6.0.3 Windows SDK with the trunk 6.2 Swift compiler. As explained there, that works fine now because the Swift 6.2 compiler likely quietly ignores the 6.0.3 resource directory in that 6.0.3 SDK and uses the 6.2 stdlib modules next to the compiler instead, but when I enforced that it must use the 6.0.3 SDK alone in that pull, the compile failed as expected. We will need to shake out all such -sdk leaks before we can rely on this flag for cross-compilation.
2. The -sdk/-resource-dir flags look directly in those paths for the C sysroot and Swift runtime resources, but when using -sdk/-sysroot instead, -sdk looks in <sdkPath>/usr/lib/swift instead for the Swift-specific files. I understand why this was done, for backward compatibility of the -sdk flag, but this means you cannot specify an arbitrary Swift resource path anymore.
   For example, when building the compiler tools themselves in this repo, you will often see the flags -sdk / -resource-dir /home/finagolfin/build/Ninja-Release/swift-linux-aarch64/lib/swift/ used to compile against the system C/C++ sysroot and the freshly-built Swift stdlib. How do you replace that with -sdk/-sysroot? We'll probably have to add a third hidden flag -sdk-runtime that works just like -resource-dir currently does, if we ever want to remove the -resource-dir flag from building this repo itself.
3. The -sysroot flag is brand new and barely used, I see a couple bugs in its implementation already. This is easiest to remedy since it's new.

Regarding 1., since all current approaches use -sdk, we have no choice but to make it better. My in-progress work on making it work properly in #79621 is shaking out more bugs, which I will file and fix as I'm able. At the very least, this suggests we might have to go slower with cross-compilation SDKs while these problems are shaken out.

@compnerd, let me know your thoughts.

@al45tair, as PSG chair, maybe you have some input.

@etcwilde, you deal with these cross-compilation issues more than most, thoughts welcome.

@MaxDesiatov, the SDK bundles feature you helped create uses the current -sdk/-resource-dir flags a lot, perhaps you have some input.

[etcwilde]

So I don't think getting rid of the -resource-dir is the goal of that proposal, though certainly lessening the reliance on setting it. The compiler resources are meant to be things that are tied to a given version of the compiler. Swift hasn't been good about keeping them separate, so I'll use clang as an example, but the sanitizer runtimes and the clang builtin headers are specific to that exact clang and thus must ship as part of the toolchain. That's what the -resource-dir should point at. For Swift, we definitely want the clang-importer headers in a subdirectory of the resource directory. Some of the shims also probably belong in that directory too like HeapObject.h and Task.h, but not all of them. The C++ bridging code isn't stable and is also tied to a specific build of the compiler, so those should also be shipped in there too. Under normal circumstances, people shouldn't really need to touch this, but stuff happens.

Under the new definition, the -sdk flag should point to the Swift runtimes for the platform you're targeting, and then the -sysroot should point at the runtimes and libraries provided by the OS. This is mostly because most OS's don't come with a Swift stdlib and don't want to install the Swift runtimes into your sysroot, or can't because you're trying to build something for your local system and don't have privileges to do so.

I haven't personally lived on the flags yet since we haven't split stuff up and things aren't done cooking in this regard yet, but this will become more important as we split the SDK distribution from the per-platform toolchain monolith that we have today. My eventual goal is to be able to download an Ubuntu sysroot, an Ubuntu Swift SDK, and cross-compile a little CLI application for Ubuntu from my Mac without having to be super clever about it (or whatever OS's we decide to produce Swift SDKs for). I'm not denying that things are probably not working right at the moment, but that's the direction we're trying to push things in.

For example, when building the compiler tools themselves in this repo, you will often see the flags -sdk / -resource-dir /home/finagolfin/build/Ninja-Release/swift-linux-aarch64/lib/swift/ used to compile against the system C/C++ sysroot and the freshly-built Swift stdlib. How do you replace that with -sdk/-sysroot? We'll probably have to add a third hidden flag -sdk-runtime that works just like -resource-dir currently does, if we ever want to remove the -resource-dir flag from building this repo itself.

In this example, -sdk points at the standard libraries (swiftCore, concurrency, etc...) and their modules. -sysroot points at the C/C++ libraries coming from the system sysroot, and -resource-dir points at the compiler resources. This will make a lot more sense once the standard library and compiler build systems are fully split from each other. The standard library build won't be installing the just-built standard library into the compiler build directory since it's a separate project, so -sdk will need to point at where the stdlib was installed to.

[finagolfin]

So I don't think getting rid of the -resource-dir is the goal of that proposal, though certainly lessening the reliance on setting it.

OK, I thought the plan was to deprecate the flag altogether, but changing the meaning of such a core compilation flag instead is going to be even harder.

Under the new definition, the -sdk flag should point to the Swift runtimes for the platform you're targeting

Just to be clear, because it is not specified in that new cross-compilation document, will the new model look directly in the specified -sdk path for the runtime modules and libraries or in -sdk/<OS> or in -sdk/usr/lib/swift/<OS>? If the last, to maintain compatibility with the old model of -sdk pointing at a C/C++ sysroot, then the new model cannot point at an arbitrary Swift resource directory anymore, such as my example above of building the compiler tools in this repo.

this will become more important as we split the SDK distribution from the per-platform toolchain monolith that we have today... This will make a lot more sense once the standard library and compiler build systems are fully split from each other.

It would be nice if whoever is planning this split would write up what files are going to go where and exactly what compiler flags will be used to access them, instead of the vague hand-waving in the current cross-compilation doc.

My eventual goal is to be able to download an Ubuntu sysroot, an Ubuntu Swift SDK, and cross-compile a little CLI application for Ubuntu from my Mac without having to be super clever about it (or whatever OS's we decide to produce Swift SDKs for).

Sounds great, but the transition process to get there looks rough.

I spent some time this weekend investigating how the limited support for an external -sdk has worked over the years, and it turns out that it only ever supported that being a C/C++ sysroot. While some limited support for it being a "full SDK," ie also containing non-Darwin Swift runtime modules and libraries in -sdk/usr/lib/swift/<OS> without also explicitly specifying that path as the -resource-dir, was added over the years to the old C++ Driver, #25990, and more recently to the new swift-driver in 6.1, swiftlang/swift-driver#1560, I found that it never really worked, because no support was ever added in the swift-frontend to look for the Swift runtime modules in -sdk/usr/lib/swift/<OS>, ie both the drivers would look in -sdk/usr/lib/swift/<OS> for a few files like swiftrt.o or libswiftCore.so but the frontend then ignores the -sdk and gets the runtime modules like Swift.swiftmodule from outside the -sdk! I'm trying to finally fix this in #79621.

Supporting such full SDKs is not too important, but I bring it up for two reasons. One is that the plan appears to be in the new cross-compilation model to look for the Swift resource directory in -sdk/usr/lib/swift, as implemented for non-Darwin Unix in swiftlang/swift-driver#1560 for swiftrt.o alone, but that has never worked for such full SDKs in the old model. The other is that by implementing that new model for just that one Unix object file, the old -sdk/-resource-dir model broke in many scenarios, ie when -sdk does not contain a Swift resource directory or just an incompatible one. I have to work around this on my Android CI ever since that swift-driver pull was merged against my objections.

I suggest you take a look at #79621 and the associated swiftlang/swift-driver#1822 and see if that fits your plans for this new cross-compilation model. Both are tiny pulls that keep both old and new cross-compilation models working, while moving the new one laid out in the cross-compilation doc forward.