Support defining UDFs in RecordMetaData

[pengpeng-lu]

No description provided.

[alecgrieser]

A few small things. Most of the "bigger" things are probably more points for discussion/follow-up work as this matures, and then the smaller things should be hopefully pretty straightforward. Sorry for the delay in getting around to this; just been a lot going on trying to get the build up and running and 4.1 validated.

[alecgrieser]

Should this throw an error instead of returning null if we don't find the correct type? It seems like it would be hard to debug a NullPointerException if we fall into this path, and it also seems like we wouldn't want to just silently ignore an unserializable field here. (It does also mean that we need to be somewhat cautious when we roll out new classes here, or new functions, but there's not a lot we can do about that.)

[pengpeng-lu]

makes sense. return a RuntimeException instead.

[alecgrieser]

I actually don't see this code any more. Did this become unnecessary after switching over to registering a deserializer with the auto service?

[alecgrieser]

Is this class Serializable? I would only expect to need the serialVersionUID if it implemented the Serializable interface, which I don't think this class should

[pengpeng-lu]

No, but I keep seeing this weird pmd error: MissingSerialVersionUID: Classes implementing Serializable should set a serialVersionUID, I checked all parent classes but couldn't find where it comes from...

[alecgrieser]

I do like this change. It will require that any user who directly interacts with this proto file update their code, though it's a pretty straightforward fix. Still, I think it would be good to target getting this into 4.2 rather than 4.1. We should be just about to finish our final 4.1 build, so we can merge this after that is ready. (The only things I'm aware of that we want to target for 4.1 are #3204 and #3207.)

[alecgrieser]

Overall looks good, though one last thing about serialization. Also, I think this should probably update gradle.properties to set the version to 4.2.0.0, which should be the only other thing necessary to bump the version to 4.2

[alecgrieser]

Sorry, this is actually something that didn't occur to me at first: is it safe to re-use the same serializationContext for multiple UDFs? Amongst other things, it contains state for values, so that if the same value is used multiple times, we only include it once (in, say, a continuation). I think that means that as written, we could end up including a value by reference across different UDFs in the same meta-data. I don't think we want that, though, because for one thing, it means that you have to be careful to serializea and deserialize these functions in the same order, and removing ones means doing a full deserialization, removing it, and then fully reserializing instead of being able to remove the value at the proto level.

[alecgrieser]

I actually don't see this code any more. Did this become unnecessary after switching over to registering a deserializer with the auto service?

[alecgrieser]

Okay, the fact that PMD didn't like this class with the old name. If we rename the protobuf to something like PUserDefinedFunction, does that fix things? The MetaData proto class doesn't follow that convention, but it would make this more in line with the planner protos where each POJO has a proto equivalent with the P prefix. I might prefer that over to including Serialized in this class's name, though in principle either could work.

[pengpeng-lu]

makes sense. changed.

[alecgrieser]

Let's merge this once https://github.com/FoundationDB/fdb-record-layer/actions/runs/13861613630 completes. The release workflow should be robust to things merging causing conflicts, but it would be nice to get a clean run first.