[Unix] Go back to only checking the runtime resource path for swiftrt.o

[finagolfin]

This is made possible by the swift-frontend now setting this instead in swiftlang/swift#79621. More changes incoming...

[finagolfin]

Ready for review along with the required linked frontend pull, which passed the linux CI when tested with this pull (this pull has no effect on Darwin and Windows). Comments should mostly go there, since this is just a consequence of that pull.

[finagolfin]

@swift-ci please test

[finagolfin]

Alright, this is ready to go, a partial revert of #1667, independent of the linked changes in swiftlang/swift#79621. This pull alone passed CI, along with a full Windows toolchain build. I'd like to get this into 6.1 also next.

Some background: the prior change checked the -sdk alone for swiftrt.o, which sometimes works if the -sdk has one and it isn't too different, but usually doesn't. Under the planned new cross-compilation model, all Swift runtime resources like libswiftCore.so will be looked for in -sdk/usr/lib/swift/<os>, but this reverted code only looks for the single file swiftrt.o, which means the runtime libraries are not looked for in the same -sdk directory! This mixing of Swift runtime resources from two different resource directories in the new model may sometimes work, but will often break.

As for the original cross-compilation model referenced in that doc, which is still the one commonly used, this change I'm reverting often breaks builds because -sdk refers to a C/C++ sysroot alone, like the Android NDK, which usually contains no Swift resources. Since the prior change is broken in both cross-compilation models, I'm reverting it and proposing swiftlang/swift#79621 instead to implement it properly for both the original and new model.

@artemcm, please review.

@compnerd, let us know what you think.

[finagolfin]

Ping @artemcm, mind approving this? I'd like to get it into 6.1 next and I don't think Saleem will respond.

Ever since I argued against this change in the linked pull last year, I have occasionally raised the problems that this caused, eg swiftlang/swift#76381, and got no response. I think they were probably experimenting with new SDK layouts at the time, tried this hack, then moved on to other things.

Let's fix this now, otherwise it is going to start breaking 6.1 SDK bundles and toolchain builds on Unix, ie the two main scenarios where an -sdk is now explicitly specified.

[finagolfin]

@etcwilde, mind reviewing this? I suspect I'm not going to get a response from those above, since they approved the original pull that broke this, and I'd like to get this fixed before it finally ships in 6.1.

[etcwilde]

swiftrt.o is associated with the SDK content though so that the sections are set up correctly for the implementation of swift_addNewDSOImage coming from the stdlib.

During the hybrid phase, this means that we should be grabbing swiftrt from the same place that we found libswiftCore (either via the sdk or resource-dir), though for expedience I'm fine with not hooking that exact logic up today.

As a compromise, can we continue to search the SDK first, and if it's not found there, fall back on the resource-dir?

[finagolfin]

"SDK" can mean different things under the different cross-compilation models, so I try to be more specific.

Most compilation does not set the -sdk flag explicitly, so this code is ignored. We primarily set it explicitly in two scenarios:

1. cross-compilation
2. Building the Swift compiler and stdlib itself from source

The problem with this previous Unix change for most cross-compilation SDK bundles is that they use the old model of -sdk having a C/C++ sysroot alone and a separate -resource-dir containing Swift modules/libraries. Despite the bundle pitch doc saying otherwise, both flags are required, because of a separate bug in this swift-driver. This code I'm changing then fails as the -sdk contains only C/C++ files and no swiftrt.o, see swiftlang/swift#76381 for a concrete example.

As for searching both, the problem is it might find a swiftrt.o in -sdk and still fail because it's the wrong version. I'm trying to fix all this more comprehensively for Swift 6.2 in swiftlang/swift#79621, but I'd like to revert this broken swift-driver code first for 6.1, as we probably won't want to rush swiftlang/swift#79621 into 6.1.

[finagolfin]

Ping @etcwilde or @artemcm, I think this should be merged before the 6.1 release, as SDK bundles and building the Swift toolchain itself with Swift 6.1 is going to start breaking on non-Darwin Unix otherwise, as shown in swiftlang/swift#76381. I get these runtime resources from an external -sdk properly in the frontend pull swiftlang/swift#79621 instead, and intend to implement this more completely that way instead.

[finagolfin]

@shahmishal, could you take a look at this, as it's been a couple weeks and I haven't really been able to get a review here? I tried building the 6.1 compiler with a prebuilt 6.1 on linux to see if the change this is reverting broke anything and it didn't, as the compiler build is careful to use clang for linking Swift executables instead of swift-driver, but the moment that changes, this previous change that I'm reverting will break the compiler build. It has already broken cross-compiling with SDKs on my Android CI with the -sdk/-resource-dir flags.

I'm asking you since you're the review manager for linux and I'd like to get this into 6.1 next.

[etcwilde]

@al45tair, I know that this will affect the static linux SDK, do you have feedback?
@compnerd, do you have feedback here? Does this break your SDKs?

As for searching both, the problem is it might find a swiftrt.o in -sdk and still fail because it's the wrong version.

This is true. But we are also moving away from the model where the SDK content is shoved in the compiler resource directory.

@compnerd, you were driving the push to split the system C sysroot flag from the SDK flag. What is missing in
swiftlang/swift#76381? At least at first glance, the compiler invocation is only using the -sdk flag and not the -sysroot flag. I would expect the situation where you're using the C runtimes from the system and a custom runtime to look something like this swiftc -sdk <path to Swift runtimes> -sysroot /.

[finagolfin]

But we are also moving away from the model where the SDK content is shoved in the compiler resource directory.

Essentially nothing has been done to make that move, other than this change looking for a single file alone, which means currently under your new model, the 6.1 compiler always looks in two different directories for Swift runtime resources like swiftrt.o or Swift.swiftmodule when a -sdk is specified, which is a recipe for compilation failures when they don't match versions.

At least at first glance, the compiler invocation is only using the -sdk flag and not the -sysroot flag. I would expect the situation where you're using the C runtimes from the system and a custom runtime to look something like this swiftc -sdk -sysroot /.

Yes, the invocation in swiftlang/swift#76381 is using the current -sdk/-resource-dir model, because the planned new -sdk/-sysroot model doesn't work yet. Particularly, a glaring hole in the new model with the current compiler implementation is that the frontend will not look in -sdk or -sysroot for the Swift stdlib modules! I detailed that for you here a couple weeks ago.

[finagolfin]

Sigh, with all the delays in reviewing this, 6.1 has now shipped with this regression, so we'll have to fix it in a patch release.