Have the frontend and new swift-driver look in an external -sdk for non-Darwin platform runtime libraries and modules too

[finagolfin]

as done originally in #25990 with the legacy C++ Driver, but since lost in the new swift-driver

This is another attempt to fix swiftlang/swift-driver#1562 and avoid the various workarounds that have been put in elsewhere to spackle over this root cause.

This worked for me with the compiler validation suite running natively on an Android device, with the exception of one SourceKit test. Let's run it through all the platforms on the CI before explaining fully what it's doing and adding a test to make sure it doesn't regress.

[finagolfin]

@swift-ci please test

[finagolfin]

swiftlang/swift-driver#1822
@swift-ci please test linux

[finagolfin]

swiftlang/swift-driver#1822
@swift-ci please test linux

[finagolfin]

I initially tried simply checking if <sdkRoot>/usr/lib/swift/, ie a Swift resource directory, existed, but around 100 tests from the compiler validation suite failed with that, as they pass in a separate -sdk that simply has a Swift resource directory with a few different API notes or something, but does not contain the platform-specific files needed in <sdkRoot>/usr/lib/swift/<os>/. Actually checking for this OS-specific directory makes sure the Swift resource directory in -sdk contains the required platform-specific modules and libraries, and falls back to the Swift resource directory in the toolchain itself otherwise. However, that implies mixing and matching files from two different Swift resource directories, so maybe it will be better to enforce that only one Swift resource directory is used and modify all those tests instead?

For an example of such mixing already taking place, the Windows CI failed with this pull, right before trying to compile the trunk Foundation macros for the Windows host:

-- Check for working Swift compiler: T:/5/bin/swiftc.exe - broken
CMake Error at C:/Program Files/CMake/share/cmake-3.29/Modules/CMakeTestSwiftCompiler.cmake:40 (message):
  The Swift compiler

    "T:/5/bin/swiftc.exe"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: 'T:/x86_64-unknown-windows-msvc/FoundationMacros/CMakeFiles/CMakeScratch/TryCompile-6u6j6e'
    
    Run Build Command(s): C:/PROGRA~1/MICROS~2/2022/COMMUN~1/Common7/IDE/COMMON~1/MICROS~1/CMake/Ninja/ninja.exe -v cmTC_bc87d
    [1/2][ 50%][0.084s] T:\5\bin\swiftc.exe -target x86_64-unknown-windows-msvc -j 36 -num-threads 36 -c  -module-name cmTC_bc87d -sdk T:/toolchains/swift-6.0.3-RELEASE-windows10/LocalApp/Programs/Swift/Platforms/6.0.3/Windows.platform/Developer/SDKs/Windows.sdk -gnone -Xlinker /INCREMENTAL:NO -Xlinker /OPT:REF -Xlinker /OPT:ICF  -incremental -output-file-map CMakeFiles\cmTC_bc87d.dir\\output-file-map.json  T:\x86_64-unknown-windows-msvc\FoundationMacros\CMakeFiles\CMakeScratch\TryCompile-6u6j6e\main.swift
    FAILED: CMakeFiles/cmTC_bc87d.dir/main.swift.obj 
    T:\5\bin\swiftc.exe -target x86_64-unknown-windows-msvc -j 36 -num-threads 36 -c  -module-name cmTC_bc87d -sdk T:/toolchains/swift-6.0.3-RELEASE-windows10/LocalApp/Programs/Swift/Platforms/6.0.3/Windows.platform/Developer/SDKs/Windows.sdk -gnone -Xlinker /INCREMENTAL:NO -Xlinker /OPT:REF -Xlinker /OPT:ICF  -incremental -output-file-map CMakeFiles\cmTC_bc87d.dir\\output-file-map.json  T:\x86_64-unknown-windows-msvc\FoundationMacros\CMakeFiles\CMakeScratch\TryCompile-6u6j6e\main.swift
    <unknown>:0: warning: using (deprecated) legacy driver, Swift installation does not contain swift-driver at: 'C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\build\5\bin\swift-driver-new.exe'
    <unknown>:0: warning: option '-incremental' is only supported in swift-driver
    <unknown>:0: error: module compiled with Swift 6.0.3 cannot be imported by the Swift 6.2 compiler: T:/toolchains/swift-6.0.3-RELEASE-windows10/LocalApp/Programs/Swift/Platforms/6.0.3/Windows.platform/Developer/SDKs/Windows.sdk\usr\lib\swift\windows\Swift.swiftmodule\x86_64-unknown-windows-msvc.swiftmodule

    ninja: build stopped: subcommand failed.

The issue appears to be that they're trying to build the trunk Foundation macros with the freshly-built trunk 6.2 Swift compiler, but passing in a Swift 6.0.3 SDK with -sdk T:/toolchains/swift-6.0.3-RELEASE-windows10/LocalApp/Programs/Swift/Platforms/6.0.3/Windows.platform/Developer/SDKs/Windows.sdk to build it. The trunk Swift 6.2 compiler then quietly ignores that 6.0.3 Swift resource directory and likely uses the 6.2 Swift resource directory next to the toolchain instead. This pull, by enforcing that the Swift resource directory in the 6.0.3 -sdk that is passed in is used because <6.0.3-sdkRoot>/usr/lib/swift/windows/ exists, then fails because the 6.0.3 modules cannot be used with the trunk Swift 6.2 compiler.

@weliveindetail, do you know if that's merely a configuration mistake when building the Foundation and Testing macros on the Windows CI, which should be easily fixed, or something more complicated?

[finagolfin]

This code explicitly follows what the original C++ Driver has long done when looking for the Swift runtime libraries, swiftrt.o, and a few other files found in the Swift resource directory:

if (const Arg *A = args.getLastArg(options::OPT_resource_dir)) {
    StringRef value = A->getValue();
    resourceDirPath.append(value.begin(), value.end());
  } else if (!getTriple().isOSDarwin() && args.hasArg(options::OPT_sdk)) {
    StringRef value = args.getLastArg(options::OPT_sdk)->getValue();
    resourceDirPath.append(value.begin(), value.end());
    llvm::sys::path::append(resourceDirPath, "usr");
    CompilerInvocation::appendSwiftLibDir(resourceDirPath, shared);
  } else {
    auto programPath = getDriver().getSwiftProgramPath();
    CompilerInvocation::computeRuntimeResourcePathFromExecutablePath(
        programPath, shared, resourceDirPath);
  }

Note how the SDK is only looked in if a non-Darwin -sdk is explicitly specified: Saleem later tried to expand that to Darwin also in #26361, but he may have never got it to work.

That C++ Driver setup now matches this Frontend setup, because the default in both is now to look relative to the compiler, which is done first in this Frontend here, ie usr/bin/../lib/swift/. If a -resource-dir is set, that is given first priority, then a non-Darwin -sdk is given second priority, ie the C++ Driver and the Frontend now match in where they look.

This is important for two reasons:

1. The new swift-driver simply queries the Frontend and uses whatever Swift resource directory it uses, so now the new swift-driver finally matches the original C++ Driver's behavior, and piecemeal workarounds like that in [Unix] Go back to only checking the runtime resource path for swiftrt.o swift-driver#1822 can now be eliminated.
2. The Frontend will now look in the same Swift resource directory for stdlib/corelibs swiftmodules as the swift-driver is looking for runtime libraries and swiftrt.o, eliminating any confusion between the two by centralizing the Swift resource directory lookup here. That already found one bug in the Windows CI, see my other code comment.

However, unlike the C++ Driver, my -sdk code below actually checks if the -sdk path contains a Swift resource directory for the platform triple and does not use the -sdk for this if not, falling back to the aforementioned default next to the compiler in that case. This is because an -sdk is not guaranteed to have a Swift resource directory and may have only a C/C++ sysroot.

We should probably tighten this up to require an explicit -sdk to have a Swift resource directory, with the only exception when an explicit -resource-dir is also specified, but I'm open to debate here. The C++ Driver doesn't even check if the non-Darwin -sdk has a Swift resource directory and simply assumes one is there, we can do a bit better than that.

[finagolfin]

Alright, this tiny pull and the resulting swiftlang/swift-driver#1822 cleanup now pass the linux CI, have no effect on Darwin, and found a seeming bug in the Windows CI. I'm going to test this more in comparison to the C++ Driver and examine more tests to see if I can use it to flush out more cross-compilation bugs with an explicit non-Darwin -sdk.

In the meantime, the basic functionality works and is ready for review. I will add tests once the few remaining smaller design choices mentioned above are hammered out.

@artemcm, @etcwilde, and @compnerd, please take a look: I'm looking for feedback on both this current patch and the remaining issues and questions in my detailed comments above.

[finagolfin]

swiftlang/swift-driver#1827
@swift-ci please build toolchain

[finagolfin]

swiftlang/swift-driver#1827
@swift-ci please test source compatibility

[finagolfin]

Alright, ran some more tests and came up with what I think is the near-final version of these compiler changes: the rest of this pull will just be updating the tests mentioned earlier to work with these changes.

Some background: if one passes in a -sdk flag to the compiler, there are three places one might expect the compiler to look for the associated Swift runtime modules and libraries:

1. You also specify an arbitrary -resource-dir explicitly that has the Swift runtime modules/libraries. This has long worked, though some recent changes in the upcoming 6.1 break it in some scenarios.
2. You have a "full SDK" that contains a Swift resource directory in <sdk>/usr/lib/swift/ and don't specify an explicit -resource-dir. This has never worked on non-Darwin platforms, which is particularly a problem since the planned new cross-compilation model explicitly relies on -sdk containing a Swift resource directory. It never worked because while the old C++ Driver looked in the -sdk for Swift runtime libraries and some other files and the upcoming 6.1 swift-driver looks there for swiftrt.o alone, the Swift frontend has never looked in -sdk for the Swift runtime modules.
3. You have a -sdk with no Swift resource directory, so the compiler looks for a Swift resource directory next to the compiler instead, ie in usr/bin/../lib/swift/. Since 2. never worked, sometimes people thought it was when it was really falling back to this, eg the current Windows CI failure detailed above.

If we're ever going to support full non-Darwin SDKs or make the new cross-compilation model work, we will have to transition the -sdk flag to require a Swift resource directory, but we will have to first smoke out all inadvertent cases of 3. and tell those devs to change that, likely to 1. That is why this pull does not require the -sdk to contain a Swift resource directory yet, but simply dumps out a long warning and defaults to 3. as always.

@compnerd and @etcwilde, time for you two to chime in, as the authors of that new cross-compilation model doc and likely the two most familiar with these cross-compilation flags. We all missed that full SDKs have always been broken for these non-Darwin platforms, time to fix it now that full SDKs are becoming more common, whether more linux distros that have Swift toolchain packages that install to /usr/ now or SDK bundles like mine or TBC's.

Once I get this to pass CI and it's in trunk, I'd like to get it into 6.1 next, because upcoming 6.1 changes like those in swiftlang/swift-driver#1822 break 1. above in several cases.