dynamic_cast failure with darwin+multiple modules+template classes+dynamic_cast

[Rconybea]

Ran into the following issue using following toolchain:

• apple m1 silicon, macos 12.4 (monterey)
• cmake 3.23.2
• clang 13.0.1
• pybind11 2.9.2 + using pybind11-provided cmake support
• python 3.9

Have a project involving multiple shared libraries, and multiple pybind11 modules. All symbols are using default visibility.

If we have the following:

// foo.hpp 
struct Base {};
template<typename T> class Foo : public Base {};

template<typename T> 
Foo<T> * unreliable_cast(Base * x) { return dynamic_cast<Foo<T>*>(x); }

module A:

#include "foo.hpp"
Base * make() { return new Foo<int>(); }

module B:

#include "foo.hpp"
bool check(Base * x) { return dynamic_cast<Foo<int>*>(x) != nullptr; }

then check(make()); succeeds if make() and check() are compiled in the same module, but fails if make() and check() are put in separate modules.

I believe the problem comes from interaction between:

1. each module defines its own copy of Foo<int> typeinfo
2. clang dynamic_cast<> implementation compares typeinfo pointers, so relies on ODR being satisfied in executable for typeinfo objects.
3. for template classes, compiler may be forced to store typeinfo in multiple places, possibly in every .o file that depends on the class declaration.
4. default "two-level namespace linking" allows typeinfo instances from separate modules to resolve separately in an executable. In particular appears to defeat coalescing of weak external references.

In my project, managed to workaround by disabling two-level namespaces.
Needed to make two kinds of change to CMakeLists.txt files:

# set linker flags for non-pybind11 shared libraries
add_link_options("$<$<PLATFORM_ID:Darwin>:LINKER:-flat_namespace>")
# or
target_link_options(mytarget PRIVATE "$<$<PLATFORM_ID:Darwin>:LINKER:-flat_namespace>")

# also need to set linker flags for pybind11 shared libraries
set_property(
  TARGET pybind11::python_link_helper
  APPEND
  PROPERTY INTERFACE_LINK_OPTIONS "$<$<PLATFORM_ID:Darwin>:LINKER:-flat_namespace>"
  )

references:
https://www.russellmcc.com/posts/2013-08-03-rtti.html#foot1
https://stackoverflow.com/questions/47322895/dynamic-cast-doesnt-work-across-module-boundaries-on-clang