Transmute between functions with different return types, where one function never returns

[Michael-F-Bryan]

A question came up on the community Discord today and I'd like to know if my reasoning was correct.

This was the original question:

Why is it impossible to cast fn() -> ! to fn() and would it be safe to perform the conversion with a transmute?

And the following conversation ensued:

Michael-F-Bryan: I think the only valid casts/coersions for a function are extending/shrinking lifetimes (e.g. between fn(&'static str) and fn<'a>(&'a str)) and non-capturing closures to function pointers.

#![cfg_attr(goat, no_std)]: yes, they are covariant over the return type
! is a subtype of all types, isn't it

Michael-F-Bryan: transmuting between them could be unsound because of things like Return Value Optimisation... Imagine you had a fn() -> [u8; 256] and LLVM decided to apply RVO, turning it into fn(&mut [u8; 256]) to elide the copy. Now you would be transmuting a function which accepts no arguments(fn() -> !) into a function that accepts one argument, and that may mess up your stack/registers depending on your calling convention.
just hypothesizing

#![cfg_attr(goat, no_std)]: the return type is ()
it's not subject to RVO
besides I think it is safe to ignore parameters in most if not all calling conventions
since you never return, you never observe side effects of failed RVO

(I then posted a playground example which runs the transmute with Miri, and Miri complains that it's UB)

#![cfg_attr(goat, no_std)]
oh
so ! is not memory layout compatible with any type

Michael-F-Bryan: I believe it's because the general case of transmuting between functions that return different things is unsound unless the returned value is a pointer
which is why it's valid for compilers to apply things like RVO
Miri was happy for me to turn a fn() -> *mut u8 into a fn() -> mut String

Is my reasoning correct that, generally, optimisations like RVO being sound is what makes this transmute is UB?

[bjorn3]

Imagine you had a fn() -> [u8; 256] and LLVM decided to apply RVO, turning it into fn(&mut [u8; 256]) to elide the copy.

That is actually already done at the abi level without involving any optimizations. The x86_64 System-V abi among other abi's pass a return value pointer to a function when the return value can't be stored in registers. This extra argument always comes first. If it were to come last, it wouldn't really be a problem. At least for most abi's. https://github.com/rust-lang/rust/blob/e261649593cf9c2707f7b30a61c46c4469c67ebb/compiler/rustc_codegen_ssa/src/mir/block.rs#L623

[digama0]

For the case of extern "C" fn(T...) -> ! and extern "C" fn(T...), at least, I would assume that the calling conventions are in a subtype relationship, because a never-returning function uses the same ABI as a unit returning function (in particular it does not pass the return outparam in the first argument). For repr(Rust) I doubt we promise that, but I don't see a particular reason not to, and more to the point it should not be UB if you guess correctly (although I can imagine Miri would have a hard time checking ABI compatibility of functions with distinct types).

[Michael-F-Bryan]

I would assume that the calling conventions are in a subtype relationship, because a never-returning function uses the same ABI as a unit returning function ... (although I can imagine Miri would have a hard time checking ABI compatibility of functions with distinct types)

It looks like Miri doesn't care about particular ABIs or subtyping and only lets you change the return type if it is a pointer. An example was made where we transmute from an extern "C" fn() -> u8 to a extern "C" fn() -> i8, which should have the same ABI shape, and Miri called it UB.

https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=2813c647acecdbd5ea0b2dbe29f5cb29

The x86_64 System-V abi among other abi's pass a return value pointer to a function when the return value can't be stored in registers. This extra argument always comes first. If it were to come last, it wouldn't really be a problem.

Thanks. I had a feeling something like this could happen, but wasn't sure of the specifics.

For the case of extern "C" fn(T...) -> ! and extern "C" fn(T...), at least, I would assume that the calling conventions are in a subtype relationship

Is the overall area of subtyping function arguments/return values something that is worth specifying for people writing unsafe code? Or would we just leave it as the status quo, "it's UB to transmute between anything other than lifetimes (respecting variance) and pointer types"?

[digama0]

I would assume that the calling conventions are in a subtype relationship, because a never-returning function uses the same ABI as a unit returning function ... (although I can imagine Miri would have a hard time checking ABI compatibility of functions with distinct types)

It looks like Miri doesn't care about particular ABIs or subtyping and only lets you change the return type if it is a pointer. An example was made where we transmute from an extern "C" fn() -> u8 to a extern "C" fn() -> i8, which should have the same ABI shape, and Miri called it UB.

I just tried that with Wrapper<u8> where #[repr(C)] struct Wrapper<T>(T); and got the same error. I think the argument for the well-definedness of this transmutation is pretty strong, so I would say that Miri is simply being too conservative here.

For the case of extern "C" fn(T...) -> ! and extern "C" fn(T...), at least, I would assume that the calling conventions are in a subtype relationship

Is the overall area of subtyping function arguments/return values something that is worth specifying for people writing unsafe code? Or would we just leave it as the status quo, "it's UB to transmute between anything other than lifetimes (respecting variance) and pointer types"?

Here I mean subtyping in the dynamic sense, i.e. a function pointer of one kind can be used in all contexts where the other one could be used. This is really what unsafe authors care about, and in fact it may not be possible for the abstract machine to be able to tell the difference between a type and a transparent wrapper type around it, so I don't think it is possible for this to be declared as UB even if we wanted to. In particular, lifetimes don't actually exist in the rust abstract machine, just "stacked borrows" stored in the memory, so you can't use any wording like "lifetimes (respecting variance)" in the definition of whether this is UB.

As a more positive proposal, I would suggest not saying anything about function pointer casting being UB; all transmutes are okay, but calling a function requires that the input ABI match the ABI of the function, and returning from a function requires that the output ABI matches as well. This may have the effect of transmuting the function parameters if the ABI matches but the types don't, and this is not UB until the transmuted parameter is used, assuming the types don't match up.

[chorman0773]

I just tried that with Wrapper where #[repr(C)] struct Wrapper(T);

The ABI considerations of those two types may be different.

I think that the original fn()->!->fn()->() should be permited, as I believe it's noted that an fn()->() pointer is ABI compatible with the C function type void(void), and that fn()->! is equivalent to _Noreturn void(void) (and because in C, _Noreturn does not affect the function type, those signatures must be compatible in C),

[Michael-F-Bryan]

I would suggest not saying anything about function pointer casting being UB; all transmutes are okay, but calling a function requires that the input ABI match the ABI of the function, and returning from a function requires that the output ABI matches as well.

This sounds like a pretty reasonable rule-of-thumb 👍

It's also easier to specify what is valid instead of listing a bunch of exceptions where something isn't valid.

Here I mean subtyping in the dynamic sense, i.e. a function pointer of one kind can be used in all contexts where the other one could be used. This is really what unsafe authors care about, and in fact it may not be possible for the abstract machine to be able to tell the difference between a type and a transparent wrapper type around it, so I don't think it is possible for this to be declared as UB even if we wanted to.

This would be a big ticket item for me.

For example, I'd really like to declare FFI bindings like fn foo(Option<NonNull<Bar>>) instead of fn foo(*mut Bar), and that would only be possible if you can guarantee Option<NonNull<Bar>> is a transparent wrapper around *mut Bar.

You can do that in FFI code at the moment and it works because of #[repr(transparent)] and niche optimisation, but whenever I've talked to people they haven't been able to confidently say it's correct and guaranteed behaviour with respect to the Rust abstract machine.

[chorman0773]

For example, I'd really like to declare FFI bindings like fn foo(Option<NonNull>) instead of fn foo(*mut Bar), and that would only be possible if you can guarantee Option<NonNull> is a transparent wrapper around *mut Bar.

This is already guaranteed, as part of Option.

[digama0]

I just tried that with Wrapper where #[repr(C)] struct Wrapper(T);

The ABI considerations of those two types may be different.

I think that the original fn()->!->fn()->() should be permited, as I believe it's noted that an fn()->() pointer is ABI compatible with the C function type void(void), and that fn()->! is equivalent to _Noreturn void(void) (and because in C, _Noreturn does not affect the function type, those signatures must be compatible in C),

I think you can substitute other types that are as layout equivalent as we have promises for, for example fn() -> ZST vs fn() -> (). If we consider all of these as distinct (following miri's lead) then even things like fn(Option<NonNull>) vs fn(*mut Bar) are UB, because we need a rule that says that fn(A, B, ...) -> R and fn(A', B', ...) -> R' are compatible if A,A', B,B', ..., R,R' are (for an appropriate sense of "compatible"), and it seems we don't have that.

[Nemo157]

#[repr(C)] struct Wrapper<T>(T); specifically is not ABI compatible with T though, you probably want #[repr(transparent)] struct Wrapper<T>(T);

[burdges]

I'd expect fn() -> [T] becomes a generator eventually ala rust-lang/rfcs#2884 but presumably via being sugar so the type itself becomes a generator and not problematic here.

[RalfJung]

(I then posted a playground example which runs the transmute with Miri, and Miri complains that it's UB)

Note that the Miri checks for this are just "whatever seemed most sensible to me at the time". I tried to make them restrictive enough so that any code that is allowed is unambiguously correct, but it is very possible that the checks could be relaxed. This requires more knowledge about calling conventions and ABIs than I have, though. ;)

[chorman0773]

It may be reasonable to adopt the C rules, which must work on any platform where the underlying ABI is used by C (and if there is no platform C abi, rust can do whatever it wants for extern"C"). Whether or not this would extend abis other than extern"C", and, in particular, if it extends to extern"Rust", would be up for debate.
C allows you to call a function via a function pointer, as long as the type of the function is compatible with a function pointer. Note that this definition determines whether the call can be performed at all, and would not determine library invariants, or permit violations of any validity invariants.
Compatibility is determined as follows (where the following describes an equivalence relation):

• A type T is compatible with itself
• In the return type position only, () is compatible with !. In all other positions they are not compatible(could also be unspecified).
• A signed integer type is compatible with it's corresponding unsigned integer type
• A type T is compatible with the type Transparent<T> where Transparent<T> is any type which is transparent over T.
• All Inhabited Zero-sized types with alignment 1 are mutually compatible.
• All of the following types are mutually compatible: NonNull<T>, Box<T>, &T, &mut T, an Option arround any of those types, *mut T, and *const T.
• Two pointer types are compatible if both pointees are Sized, or the pointee types are compatible
• Two slice types are compatible if the elements are compatible
• Two structure types are compatible if they are both repr(C) structs, with the same number of members, and
• Two union types are compatible if they are both repr(C) unions, with the same number of members, and each corresponding member has a compatible type. Note - the order of declaration of union fields is not considered for compatibility - End Note
• A fundamental integer type T is compatible with NonZero<T>, where NonZero<T> is the corresponding NonZeroU* or NonZeroI* declared in core::num, and with Option<NonZero<T>>
• bool is compatible with u8
• An empty enum is compatible with !
• Two function pointer types are compatible if both have the same ABI, both have the same number of parameters, where the corresponding parameters in declaration order are compatible, and the return types are compatible.
• A function pointer type F is compatible with Option<F>.
• An array type of length 1 is compatible with it's element type.
• Two types T and U are compatible if there exists a type S, such that T is compatible with S, and S is compatible with U.

[RalfJung]

@chorman0773 that list is a great starting point, thanks!

In the return type position only, () is compatible with !. In all other positions they are not compatible(could also be unspecified).

C doesn't really have uninhabited types, so I am not entirely sure about this. You talked about _NoReturn above, but I am not sure if that is a fully adequate model of !.

By "in return type position", do you also mean nested positions in the return type? Like, is (!, ()) compatible with ((),()) as a return type?

A type T is compatible with the type Transparent where Transparent is any type which is transparent over T.

This seems to be doing a lot, and is getting close to the discussions around Wrapper<T> vs Wrapper<U>. In particular, "transparent over" is not defined.

---

I should also point at rust-lang/rust#56166: rustc's FnType encodes a lot (all?) of the information that is relevant for ABIs, so ideally it would be possible to have a check that is based solely on that. Having to do a type-based comparison would be somewhat unprecedented (except for validity, which is all about the type); I feel like this is closer to layouts and transmute, where everything Miri needs to know is determined by rustc's TyAndLayout.

[chorman0773]

You talked about _NoReturn above, but I am not sure if that is a fully adequate model of !.

Well, in the return type position, I believe they are fundamentally equivalent. A function returning ! does not return (because it cannot return a value of type !, as no such value exists). Likewise, a function that is _Noreturn in C does not return, for a different reason (the fact that returning from a _Noreturn function is UB). I know that I have, in the past, relied on ->! being the same as _Noreturn void(..)/__attribute__((noreturn)) void(..). I'm sure others have as well.

By "in return type position", do you also mean nested positions in the return type? Like, is (!, ()) compatible with ((),()) as a return type?

Honestly, I do not know. My intention is that only in the top-level return-type position, that is, fn()->(), and fn()->!. However, it may be reasonable for things like fn()->UnsafeCell<()> and fn()->UnsafeCell<!> to be compatible. Right now, the example you provided is not compatible, because no compatibility is given to tuples (intentionally). One thing I want to do is exclude compatibility in the parameter position, so that a function that accepts a parameter of an uninhabited type can be converted into an uncallable function (which has a completely distinct ABI from one where all occurances of ! were replaced with ()).

This seems to be doing a lot, and is getting close to the discussions around Wrapper<T> vs Wrapper<U>. In particular, "transparent over" is not defined.

This rule is derived, it's just written for completeness. By definition, a transparent type has the same abi as it's transparent field, so they must be compatible types. For "transparent over" that's fine, it can be defined as the field of the transparent struct which is not a 1-ZST, or () if no such field exists. Additionally, MaybeUninit<T> is transparent over T by definition as well.

I feel like this is closer to layouts and transmute, where everything Miri needs to know is determined by rustc's TyAndLayout.

This was not an accident. The assumption is that if you pass in a parameter which is the wrong type, it gets transmuted into the correct one.

[RalfJung]

This was not an accident. The assumption is that if you pass in a parameter which is the wrong type, it gets transmuted into the correct one.

Oh, sure. I am just saying, instead of having to implement your list, maybe it would also work for Miri to just check if the mode and size of all arguments and the return type match.

This rule is derived, it's just written for completeness. By definition, a transparent type has the same abi as it's transparent field

Well, a Transparent<T> could still be something like

#[repr(transparent)]
struct Transparent<T: Trait>(T::Assoc);

But I think this is just a matter of notation -- your use of generic notation <...> here seems suboptimal, but if all you want to say is that repr(transparent) structs are compatible with the type of their non-ZST field, that makes sense.