[SR-5779] SwiftRuntimeTests target fails on FreeBSD due to a linker error

[dcci]

Previous ID	SR-5779
Radar	None
Original Reporter	@dcci
Type	Bug

Additional Detail from JIRA

Votes	0
Component/s
Labels	Bug
Assignee	None
Priority	Medium

md5: 041350be3b8c017938f67d832db77dcf

Issue Description:

Now that I have some time, I'm trying to revamp the FreeBSD support in swift.

I'm seeing a bunch of linker errors when running basic tests (specifying `-t` to `build-script`).

FAILED: unittests/runtime/SwiftRuntimeTests
: && /usr/bin/clang++ Wno-unknown-warning-option -Werror=unguarded-availability-new -fno-stack-protector -fPIC -fvisibility-inlines-hidden -Werror=date-time -std=c+11 -Wall -W -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wcovered-switch-default -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fcolor-diagnostics -ffunction-sections -fdata-sections -Werror=switch -Wdocumentation -Wimplicit-fallthrough -Wunreachable-code -Woverloaded-virtual -DOBJC_OLD_DISPATCH_PROTOTYPES=0 -O2 -Wl,-O3 -Wl,-gc-sections -fuse-ld=gold stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/AnyHashableSupport.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/Casting.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/CygwinPort.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/Demangle.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/Enum.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/ErrorObjectConstants.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/ErrorObjectNative.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/Errors.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/ErrorDefaultImpls.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/Exclusivity.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/Heap.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/HeapObject.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/ImageInspectionMachO.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/ImageInspectionELF.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/ImageInspectionWin32.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/KnownMetadata.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/Metadata.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/MetadataLookup.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/MutexPThread.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/MutexWin32.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/Once.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/Portability.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/ProtocolConformance.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/RefCount.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/RuntimeEntrySymbols.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/_///lib/Demangling/OldDemangler.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir////lib/Demangling/Demangler.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir////lib/Demangling/NodePrinter.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir////lib/Demangling/Context.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir////lib/Demangling/ManglingUtils.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir////lib/Demangling/Punycode.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/ErrorObject.mm.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/SwiftObject.mm.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/SwiftValue.mm.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir/Reflection.mm.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir////lib/Demangling/OldRemangler.cpp.o stdlib/public/runtime/CMakeFiles/swiftRuntime-freebsd-x86_64.dir////lib/Demangling/Remangler.cpp.o unittests/runtime/CMakeFiles/SwiftRuntimeTests.dir/Exclusivity.cpp.o unittests/runtime/CMakeFiles/SwiftRuntimeTests.dir/Metadata.cpp.o unittests/runtime/CMakeFiles/SwiftRuntimeTests.dir/Mutex.cpp.o unittests/runtime/CMakeFiles/SwiftRuntimeTests.dir/Enum.cpp.o unittests/runtime/CMakeFiles/SwiftRuntimeTests.dir/Refcounting.cpp.o unittests/runtime/CMakeFiles/SwiftRuntimeTests.dir/Stdlib.cpp.o -o unittests/runtime/SwiftRuntimeTests -L/usr/home/davide/build/Ninja-RelWithDebInfoAssert/llvm-freebsd-x86_64/./lib -Wl,-rpath,/usr/home/davide/build/Ninja-RelWithDebInfoAssert/llvm-freebsd-x86_64/./lib:/usr/home/davide/build/Ninja-RelWithDebInfoAssert/swift-freebsd-x86_64/lib/swift/freebsd/x86_64 /usr/home/davide/build/Ninja-RelWithDebInfoAssert/llvm-freebsd-x86_64/lib/libLLVMSupport.a -lpthread /usr/home/davide/build/Ninja-RelWithDebInfoAssert/llvm-freebsd-x86_64/lib/libgtest_main.a /usr/home/davide/build/Ninja-RelWithDebInfoAssert/llvm-freebsd-x86_64/lib/libgtest.a -lpthread lib/swift/freebsd/x86_64/libswiftCore.so /usr/lib/libexecinfo.so /usr/home/davide/build/Ninja-RelWithDebInfoAssert/llvm-freebsd-x86_64/lib/libLLVMSupport.a -lrt /usr/lib/libexecinfo.so -ltinfo -lpthread -lz -lm /usr/home/davide/build/Ninja-RelWithDebInfoAssert/llvm-freebsd-x86_64/lib/libLLVMDemangle.a -lpthread && : /usr/include/c/v1/atomic:893: error: undefined reference to 'sync_val_compare_and_swap_16' /usr/include/c/v1/atomic:893: error: undefined reference to 'sync_val_compare_and_swap_16' /usr/include/c/v1/atomic:893: error: undefined reference to 'sync_val_compare_and_swap_16' /usr/include/c/v1/atomic:893: error: undefined reference to 'sync_val_compare_and_swap_16' /usr/include/c/v1/atomic:887: error: undefined reference to 'sync_lock_test_and_set_16' /usr/include/c/v1/atomic:887: error: undefined reference to 'sync_lock_test_and_set_16' /usr/include/c+/v1/atomic:887: error: undefined reference to '_sync_lock_test_and_set_16'

[dcci]

Some other informations, for completeness:

Swift Revision:

commit 8675c8e719e0896a4a37a6fdd87fafb4a2ea5283
Merge: f52b19238c 7fd9e0bb04
Author: swift-ci <swift-ci@users.noreply.github.com>
Date: Sun Aug 27 12:47:35 2017 -0700

Merge pull request #&#8203;11650 from salmanjamil/patch-1

Clang version on the host:

FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
Target: x86_64-unknown-freebsd11.1
Thread model: posix
InstalledDir: /usr/bin

uname:

FreeBSD mercat6.netperf.freebsd.org 11.1-PRERELEASE FreeBSD 11.1-PRERELEASE #​0: Fri May 26 04:11:44 UTC 2017     root@mercat6.netperf.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64

[dcci]

I'm under the impression this may be a variation of:
https://bugs.llvm.org/show_bug.cgi?id=31620#c8

I'll try to get that sorted out in LLVM, but I wonder if there's something that can be done on the swift-side to workaround this problem for the time being.

[belkadan]

Maybe -latomic, like you're using over there? It'd be unfortunate for every Swift library to depend on libatomic, though.

[dcci]

Hi Jordan.
No, I'm afraid that won't help (I tried yesterday).
The issue is that FreeBSD doesn't really ship with anything GCC whatsoever anymore, so there's no such thing as `__sync*`

[belkadan]

Hm, maybe it's supposed to be something in compiler-rt then. If it really is just the SwiftRuntimeTests target, it doesn't seem like trouble to add an extra library to link against on BSD systems.

[swift-ci]

Comment by Zac (JIRA)

I'm having the same problem but in libswiftCore.so.

Both _*sync_lock_test_and_set_16 and *_sync_val_compare_and_swap_16 are emitted as externally imported symbols but are neither are available as I don't have libstdc++ on my platform.

[swift-ci]

Comment by Zac (JIRA)

I've done some investigating. It comes from SwiftShims/HeapObjects.cpp calls to compare_exchange_weak on the C++ atomic wrapper around refCounts. I've been digging deep and it seems like TargetLoweringBase.cpp in LLVM is emitting a library call for the atomic swap and compare instead of using some builtin to try and do it with RTLIB::Libcall.

/code/xxx/buildtools/linux-x64/clang/bin/lld: error: undefined symbol: __sync_val_compare_and_swap_16
>>> referenced by atomic:926 (/code/xxx/out/../buildtools/linux-x64/clang/lib/x86_64-xxx/include/c++/v1/atomic:926)
>>> HeapObject.cpp.o:(swift::RefCounts<swift::SideTableRefCountBits>::tryIncrement()) in archive /code/swift-source/build/Ninja-DebugAssert/swift-linux-x86_64/lib/swift_static/xxx/libswiftCore.a

RTLIB::Libcall is called in two places and I'm not sure which is causing this bug. Looking into LLVM LegalizeDAG.cpp calls ConvertNodeToLibcall which uses RTLIB::Libcall. Also LegalizeIntegerTypes.cpp's DAGTypeLegalizer::ExpandIntegerResult also calls RTLIB::Libcall on this case.

[swift-ci]

Comment by Zac (JIRA)

I figured out what it is. The cmake build scripts for Linux and other archs does not set a target arch when building libswiftCore. Clang when building files with <atomic> will assume that CMPXCHG16B is not available on your target intel arch because some random old AMDs did not have it and will force libcall for those intrinsics. On Linux this more just accidently resolves to a library that is emitting libatomics methods dynamically (probably libstdc++) and libswiftCore resolves to it. On a system without any GNU stack (like you are using compiler-rt) you are screwed.

You have to change the swift built to add -mc16 as a cflag (or a -march that implies that feature is supported) when building swift libraries which will fix everything.

[dcci]

> On a system without any GNU stack (like you are using compiler-rt) you are screwed.
Pretty much.
So, something like
`SET ( CMAKE_CXX_FLAGS "-mc16" )` should do it, right?
I think it's safe enough to enable this on a per-OS basis (probably FreeBSD, as GPL-free, and maybe your platform?)

[dcci]

Can you try applying this to your local LLVM checkout and see if if fixes the problem?
https://reviews.llvm.org/D38046

Thanks,

--
Davide