bevy_reflect: Function reflection

[MrGVSV]

Objective

We're able to reflect types sooooooo... why not functions?

The goal of this PR is to make functions callable within a dynamic context, where type information is not readily available at compile time.

For example, if we have a function:

fn add(left: i32, right: i32) -> i32 {
  left + right
}

And two Reflect values we've already validated are i32 types:

let left: Box<dyn Reflect> = Box::new(2_i32);
let right: Box<dyn Reflect> = Box::new(2_i32);

We should be able to call add with these values:

// ?????
let result: Box<dyn Reflect> = add.call_dynamic(left, right);

And ideally this wouldn't just work for functions, but methods and closures too!

Right now, users have two options:

1. Manually parse the reflected data and call the function themselves
2. Rely on registered type data to handle the conversions for them

For a small function like add, this isn't too bad. But what about for more complex functions? What about for many functions?

At worst, this process is error-prone. At best, it's simply tedious.

And this is assuming we know the function at compile time. What if we want to accept a function dynamically and call it with our own arguments?

It would be much nicer if bevy_reflect could alleviate some of the problems here.

Solution

Introducing function reflection!

This adds a DynamicFunction type to wrap a function dynamically. This can be called with an ArgList, which is a dynamic list of Reflect-containing Arg arguments. It returns a FunctionResult which indicates whether or not the function call succeeded, returning a Reflect-containing Return type if it did succeed.

Many functions can be converted into this DynamicFunction type thanks to the IntoFunction trait.

Taking our previous add example, this might look something like (explicit types added for readability):

fn add(left: i32, right: i32) -> i32 {
  left + right
}

let mut function: DynamicFunction = add.into_function();
let args: ArgList = ArgList::new().push_owned(2_i32).push_owned(2_i32);
let result: Return = function.call(args).unwrap();
let value: Box<dyn Reflect> = result.unwrap_owned();
assert_eq!(value.take::<i32>().unwrap(), 4);

And it also works on closures:

let add = |left: i32, right: i32| left + right;

let mut function: DynamicFunction = add.into_function();
let args: ArgList = ArgList::new().push_owned(2_i32).push_owned(2_i32);
let result: Return = function.call(args).unwrap();
let value: Box<dyn Reflect> = result.unwrap_owned();
assert_eq!(value.take::<i32>().unwrap(), 4);

As well as methods:

#[derive(Reflect)]
struct Foo(i32);

impl Foo {
  fn add(&mut self, value: i32) {
    self.0 += value;
  }
}

let mut foo = Foo(2);

let mut function: DynamicFunction = Foo::add.into_function();
let args: ArgList = ArgList::new().push_mut(&mut foo).push_owned(2_i32);
function.call(args).unwrap();
assert_eq!(foo.0, 4);

Limitations

While this does cover many functions, it is far from a perfect system and has quite a few limitations. Here are a few of the limitations when using IntoFunction:

1. The lifetime of the return value is only tied to the lifetime of the first argument (useful for methods). This means you can't have a function like (a: i32, b: &i32) -> &i32 without creating the DynamicFunction manually.
2. Only 15 arguments are currently supported. If the first argument is a (mutable) reference, this number increases to 16.
3. Manual implementations of Reflect will need to implement the new FromArg, GetOwnership, and IntoReturn traits in order to be used as arguments/return types.

And some limitations of DynamicFunction itself:

1. All arguments share the same lifetime, or rather, they will shrink to the shortest lifetime.
2. Closures that capture their environment may need to have their DynamicFunction dropped before accessing those variables again (there is a DynamicFunction::call_once to make this a bit easier)
3. All arguments and return types must implement Reflect. While not a big surprise coming from bevy_reflect, this implementation could actually still work by swapping Reflect out with Any. Of course, that makes working with the arguments and return values a bit harder.
4. Generic functions are not supported (unless they have been manually monomorphized)

And general, reflection gotchas:

1. &str does not implement Reflect. Rather, &'static str implements Reflect (the same is true for &Path and similar types). This means that &'static str is considered an "owned" value for the sake of generating arguments. Additionally, arguments and return types containing &str will assume it's &'static str, which is almost never the desired behavior. In these cases, the only solution (I believe) is to use &String instead.

Followup Work

This PR is the first of two PRs I intend to work on. The second PR will aim to integrate this new function reflection system into the existing reflection traits and TypeInfo. The goal would be to register and call a reflected type's methods dynamically.

I chose not to do that in this PR since the diff is already quite large. I also want the discussion for both PRs to be focused on their own implementation.

Another followup I'd like to do is investigate allowing common container types as a return type, such as Option<&[mut] T> and Result<&[mut] T, E>. This would allow even more functions to opt into this system. I chose to not include it in this one, though, for the same reasoning as previously mentioned.

Alternatives

One alternative I had considered was adding a macro to convert any function into a reflection-based counterpart. The idea would be that a struct that wraps the function would be created and users could specify which arguments and return values should be Reflect. It could then be called via a new Function trait.

I think that could still work, but it will be a fair bit more involved, requiring some slightly more complex parsing. And it of course is a bit more work for the user, since they need to create the type via macro invocation.

It also makes registering these functions onto a type a bit more complicated (depending on how it's implemented).

For now, I think this is a fairly simple, yet powerful solution that provides the least amount of friction for users.

---

Showcase

Bevy now adds support for storing and calling functions dynamically using reflection!

// 1. Take a standard Rust function
fn add(left: i32, right: i32) -> i32 {
  left + right
}

// 2. Convert it into a type-erased `DynamicFunction` using the `IntoFunction` trait
let mut function: DynamicFunction = add.into_function();
// 3. Define your arguments from reflected values
let args: ArgList = ArgList::new().push_owned(2_i32).push_owned(2_i32);
// 4. Call the function with your arguments
let result: Return = function.call(args).unwrap();
// 5. Extract the return value
let value: Box<dyn Reflect> = result.unwrap_owned();
assert_eq!(value.take::<i32>().unwrap(), 4);

Changelog

TL;DR

• Added support for function reflection
• Added a new Function Reflection example:

  bevy/examples/reflection/function_reflection.rs

  Lines 1 to 157 in ba72789

  	//! This example demonstrates how functions can be called dynamically using reflection.
  	//!
  	//! Function reflection is useful for calling regular Rust functions in a dynamic context,
  	//! where the types of arguments, return values, and even the function itself aren't known at compile time.
  	//!
  	//! This can be used for things like adding scripting support to your application,
  	//! processing deserialized reflection data, or even just storing type-erased versions of your functions.
  	use bevy::reflect::func::args::ArgInfo;
  	use bevy::reflect::func::{
  	ArgList, DynamicFunction, FunctionInfo, IntoFunction, Return, ReturnInfo,
  	};
  	use bevy::reflect::Reflect;
  	// Note that the `dbg!` invocations are used purely for demonstration purposes
  	// and are not strictly necessary for the example to work.
  	fn main() {
  	// There are times when it may be helpful to store a function away for later.
  	// In Rust, we can do this by storing either a function pointer or a function trait object.
  	// For example, say we wanted to store the following function:
  	fn add(left: i32, right: i32) -> i32 {
  	left + right
  	}
  	// We could store it as either of the following:
  	let fn_pointer: fn(i32, i32) -> i32 = add;
  	let fn_trait_object: Box<dyn Fn(i32, i32) -> i32> = Box::new(add);
  	// And we can call them like so:
  	let result = fn_pointer(2, 2);
  	assert_eq!(result, 4);
  	let result = fn_trait_object(2, 2);
  	assert_eq!(result, 4);
  	// However, you'll notice that we have to know the types of the arguments and return value at compile time.
  	// This means there's not really a way to store or call these functions dynamically at runtime.
  	// Luckily, Bevy's reflection crate comes with a set of tools for doing just that!
  	// We do this by first converting our function into the reflection-based `DynamicFunction` type
  	// using the `IntoFunction` trait.
  	let mut function: DynamicFunction = dbg!(add.into_function());
  	// This time, you'll notice that `DynamicFunction` doesn't take any information about the function's arguments or return value.
  	// This is because `DynamicFunction` checks the types of the arguments and return value at runtime.
  	// Now we can generate a list of arguments:
  	let args: ArgList = dbg!(ArgList::new().push_owned(2_i32).push_owned(2_i32));
  	// And finally, we can call the function.
  	// This returns a `Result` indicating whether the function was called successfully.
  	// For now, we'll just unwrap it to get our `Return` value,
  	// which is an enum containing the function's return value.
  	let return_value: Return = dbg!(function.call(args).unwrap());
  	// The `Return` value can be pattern matched or unwrapped to get the underlying reflection data.
  	// For the sake of brevity, we'll just unwrap it here and downcast it to the expected type of `i32`.
  	let value: Box<dyn Reflect> = return_value.unwrap_owned();
  	assert_eq!(value.take::<i32>().unwrap(), 4);
  	// The same can also be done for closures.
  	let mut count = 0;
  	let increment = |amount: i32| {
  	count += amount;
  	};
  	let increment_function: DynamicFunction = dbg!(increment.into_function());
  	let args = dbg!(ArgList::new().push_owned(5_i32));
  	// `DynamicFunction`s containing closures that capture their environment like this one
  	// may need to be dropped before those captured variables may be used again.
  	// This can be done manually with `drop` or by using the `Function::call_once` method.
  	dbg!(increment_function.call_once(args).unwrap());
  	assert_eq!(count, 5);
  	// As stated before, this works for many kinds of simple functions.
  	// Functions with non-reflectable arguments or return values may not be able to be converted.
  	// Generic functions are also not supported.
  	// Additionally, the lifetime of the return value is tied to the lifetime of the first argument.
  	// However, this means that many methods (i.e. functions with a `self` parameter) are also supported:
  	#[derive(Reflect, Default)]
  	struct Data {
  	value: String,
  	}
  	impl Data {
  	fn set_value(&mut self, value: String) {
  	self.value = value;
  	}
  	// Note that only `&'static str` implements `Reflect`.
  	// To get around this limitation we can use `&String` instead.
  	fn get_value(&self) -> &String {
  	&self.value
  	}
  	}
  	let mut data = Data::default();
  	let mut set_value = dbg!(Data::set_value.into_function());
  	let args = dbg!(ArgList::new().push_mut(&mut data)).push_owned(String::from("Hello, world!"));
  	dbg!(set_value.call(args).unwrap());
  	assert_eq!(data.value, "Hello, world!");
  	let mut get_value = dbg!(Data::get_value.into_function());
  	let args = dbg!(ArgList::new().push_ref(&data));
  	let return_value = dbg!(get_value.call(args).unwrap());
  	let value: &dyn Reflect = return_value.unwrap_ref();
  	assert_eq!(value.downcast_ref::<String>().unwrap(), "Hello, world!");
  	// Lastly, for more complex use cases, you can always create a custom `DynamicFunction` manually.
  	// This is useful for functions that can't be converted via the `IntoFunction` trait.
  	// For example, this function doesn't implement `IntoFunction` due to the fact that
  	// the lifetime of the return value is not tied to the lifetime of the first argument.
  	fn get_or_insert(value: i32, container: &mut Option<i32>) -> &i32 {
  	if container.is_none() {
  	*container = Some(value);
  	}
  	container.as_ref().unwrap()
  	}
  	let mut get_or_insert_function = dbg!(DynamicFunction::new(
  	|mut args, info| {
  	let container_info = &info.args()[1];
  	let value_info = &info.args()[0];
  	// The `ArgList` contains the arguments in the order they were pushed.
  	// Therefore, we need to pop them in reverse order.
  	let container = args
  	.pop()
  	.unwrap()
  	.take_mut::<Option<i32>>(container_info)
  	.unwrap();
  	let value = args.pop().unwrap().take_owned::<i32>(value_info).unwrap();
  	Ok(Return::Ref(get_or_insert(value, container)))
  	},
  	FunctionInfo::new()
  	// We can optionally provide a name for the function
  	.with_name("get_or_insert")
  	// Since our function takes arguments, we MUST provide that argument information.
  	// The arguments should be provided in the order they are defined in the function.
  	// This is used to validate any arguments given at runtime.
  	.with_args(vec![
  	ArgInfo::new::<i32>(0).with_name("value"),
  	ArgInfo::new::<&mut Option<i32>>(1).with_name("container"),
  	])
  	// We can optionally provide return information as well.
  	.with_return_info(ReturnInfo::new::<&i32>()),
  	));
  	let mut container: Option<i32> = None;
  	let args = dbg!(ArgList::new().push_owned(5_i32).push_mut(&mut container));
  	let value = dbg!(get_or_insert_function.call(args).unwrap()).unwrap_ref();
  	assert_eq!(value.downcast_ref::<i32>(), Some(&5));
  	let args = dbg!(ArgList::new().push_owned(500_i32).push_mut(&mut container));
  	let value = dbg!(get_or_insert_function.call(args).unwrap()).unwrap_ref();
  	assert_eq!(value.downcast_ref::<i32>(), Some(&5));
  	}

Details

Added the following items:

• ArgError enum
• ArgId enum
• ArgInfo struct
• ArgList struct
• Arg enum
• DynamicFunction struct
• FromArg trait (derived with derive(Reflect))
• FunctionError enum
• FunctionInfo struct
• FunctionResult alias
• GetOwnership trait (derived with derive(Reflect))
• IntoFunction trait (with blanket implementation)
• IntoReturn trait (derived with derive(Reflect))
• Ownership enum
• ReturnInfo struct
• Return enum

[MrGVSV]

Hm, I'm debating on whether or not I should rename Function to Func. If we ever want to add a Function trait, then it might be good to do it now in order to avoid a rename. Or perhaps DynamicFunction would be a clearer name? 🤔

[cBournhonesque]

I have a few comments but overall this looks good to me!

The example at the end is fantastic, and overall the new functionality is impressive

[cBournhonesque]

This has to be &'static because the output's 'from_arg could be any lifetime?
Could we have an impl for any lifetime longer than 'from_arg?

[MrGVSV]

'from_arg will take the lifetime of the argument, so it could be any lifetime, including 'static. The &'static is just because we don't actually care about the lifetime of Self. I think you could even do impl<'a> FromArg for &'a Foo, but again we don't care about that lifetime, just the one for Item.

[cBournhonesque]

So this is why all the arguments need to have the same lifetime? Maybe it would be worth mentioning in a comment?

[MrGVSV]

Pretty much. In one iteration I had originally used an enum in an effort to avoid Vec allocations:

enum ArgList<'a> {
  Arg0,
  Arg1(Arg<'a>),
  Arg2(Arg<'a>, Arg<'a>),
  Arg3(Arg<'a>, Arg<'a>, Arg<'a>),
  // ...
  Variadic(Vec<Arg<'a>>)
}

If we did that, we potentially could maintain some degree of lifetimes by just adding a bunch of lifetimes to ArgList:

enum ArgList<'a, 'b, 'c, 'default: 'a + 'b + 'c> {
  Arg0,
  Arg1(Arg<'a>),
  Arg2(Arg<'a>, Arg<'b>),
  Arg3(Arg<'a>, Arg<'b>, Arg<'c>),
  // ...
  Variadic(Vec<Arg<'default>>)
}

The output would still only be able to tie itself to the first argument's lifetime, but doing this would mean that we shouldn't need to shrink lifetimes down (most of the time).

So that might be something to explore.

Thoughts?

[cBournhonesque]

I think the enum ArgList might work; and it wouldn't even require the Variadic variant no?
Every new arg added would update the ArgList from ArgN to ArgN+1.

Might be something to explore, but I think it's also ok to merge the functionality as is and explore this in a future PR

[MrGVSV]

I think the enum ArgList might work; and it wouldn't even require the Variadic variant no?

The Variadic would be needed so users could supply more arguments than we have variants for. While IntoFunction supports a limited number of arguments, users are still able to construct their Function manually, with as many arguments as they want.

Might be something to explore, but I think it's also ok to merge the functionality as is and explore this in a future PR

Yeah that might be a good idea. Any breakage would be minimal and really only apply to cases where the lifetime of ArgList is explicitly set by a user.

[MrGVSV]

Yeah definitely going to save this for a followup PR!

[alice-i-cecile]

Yeah, please leave this to followup.

[MrGVSV]

Updated the PR description with the new DynamicFunction naming.

[MrGVSV]

Two updates:

1. I realized that std::any::type_name could be used to automatically infer the function name, so I went ahead and made that the default
2. DynamicFunction now implements IntoFunction, allowing it to be used interchangeably with an actual function pointer or closure wherever IntoFunction is expected

[MrGVSV]

Talked about this on Discord with @NthTensor, but I think I may explore an IntoClosure/DynamicClosure split in a followup PR. This will make using a function registry a lot easier as we won't have to require a mutable reference in order to call purely-static functions.