Add Boolean Operations on Meshes

[snailtomatoes]

What problem does this solve or what need does it fill?

Adding boolean operations should enable for an ergonomic foundation for constructive geometry

What solution would you like?

Meshes should have a function, similar in usage as merge(), like so:

base_mesh.intersection(operand_mesh);
base_mesh.union(operand_mesh);
base_mesh.subtraction(operand_mesh;
base_mesh.XOR(operand_mesh);

• union or subtraction should consume operand_mesh modifyng base_mesh inplace.
• intersection and XOR should consume both meshes to return an array or list of resulting meshes.

What alternative(s) have you considered?

Boolean operations are resource intesive and a solution that's not going to be closely integrated in the current mesh handling would degrade performance even further.
In addition having tried to do so by myself anyway it's a feature too big to tackle by myself at the moment.

[alice-i-cecile]

I expect that this would be best tackled as a working group, but IMO this is desirable and within scope :)

[ethereumdegen]

I would be pleased to help with this -- i really want to see this happen .

Here is a related talk I found. I think we could make a crate for this named "bevy_csg' or something and its responsibility would be these meshing operations. But also it could allow one to spawn an editable object in a bevy world with tools to edit it .

https://www.youtube.com/watch?v=Iqmg4gblreo

Also maybe structs for the lib could be laid out like this ?? (naive)

https://github.com/ethereumdegen/degen_csg/blob/main/src/csg.rs

an MVP really only needs cuboids to start with as that would solve 99% of level-block-out applications. Maybe cylinders too. But i agree the hardest part is going to be the boolean operations on mesh vertices/faces. If we had that, CSG wouldnt be that hard to build on top.

Btw watch that youtube video starting at 10 minutes. That kind of helps it click

[ethereumdegen]

I think the most simple naive implementation is to build a fn that can take a cuboid and subtract another cuboid.

Basically what you have to do is think in terms of faces. You need to remove certain faces from the base mesh and add faces from the applying mesh.

The tricky part is you kind of need to do a loopcut tool operation on the base mesh by using the planes of the applying mesh. Im sure having some intermediate data format (not just vertices and indices) will be extremely helpful here .

However the more i read about this the more people say how complex it can get and to use existing libraries. Of course many exist in cpp

here is a rust one that is unfinished https://github.com/carlmartus/rscsg

and another https://github.com/dima634/baby_shark

[mockersf]

CSG is very nice, but it's not general mesh boolean ops

The idea with CSG is that you start with base objects that you know how to define mathematically

It's also implemented in https://noumenal.app with Bevy

[ethereumdegen]

Yeah but that is closed source so entirely useless
yes its confusing there are many terms so we need to work through that as well. In any case csg uses those operations

[mintlu8]

union or subtraction should consume operand_mesh modifyng base_mesh inplace.

Just want to point out there is no need to consume the mesh, taking a reference is just as performant.

[HackerFoo]

union or subtraction should consume operand_mesh modifyng base_mesh inplace.

Just want to point out there is no need to consume the mesh, taking a reference is just as performant.

No, it's not, because you'll end up doing a lot of cloning, assuming the mesh is first converted to some sort of spatial tree. Unless you mean a mutable reference, which is essentially the same thing, but much harder to manage in this case.

[HackerFoo]

Yeah but that is closed source so entirely useless yes its confusing there are many terms so we need to work through that as well. In any case csg uses those operations

It can't hurt to have someone around who has been researching and working on this problem for a long time. But for the same reason, I'm not willing to just hand my work over, and it's unlikely anyone else would be able to contribute in any meaningful way without spending nearly as much time studying the underlying algorithms. I don't mind sharing references that I've used, and I track other algorithms and implementations.

I suggest using an existing robust implementation such as elalish's Manifold; he has spent a long time working on his implementation as well. Unless there is someone with the background needed and willing to do this kind of work.

[RCC-CCN]

I agree, I'd like to add to the resources InteractiveAndRobustMeshBooleans.
It's an university project in C++ with many caveats but the general step used seems to be straightforward, implementation on the other hand can be confusing.

[mintlu8]

union or subtraction should consume operand_mesh modifyng base_mesh inplace.

Just want to point out there is no need to consume the mesh, taking a reference is just as performant.

No, it's not, because you'll end up doing a lot of cloning, assuming the mesh is first converted to some sort of spatial tree. Unless you mean a mutable reference, which is essentially the same thing, but much harder to manage in this case.

I meant the right hand side mesh if there's any misunderstanding.

fn union(self, rhs: &Mesh) // or &mut self ?

It is because data are already in Vecs in both meshes. There's no way to merge vecs in rust.

[bugsweeper]

I don't think we should use another project written in C++ (I don't hate C++, I was C++ developer), I like that Bevy built in Rust and FFI usage will make it dirty.

This task is not so complex if we divide it to a set of simple tasks (I put here list of naive task list. It based on expectation that mesh is complete and has not gaps. Resulting mesh should also be complete with no gaps):

• Determine whether the point is inside of the mesh. This can be done by simply set axis (for example Z) through the point and find the most close to the point triangles crossing this axis from two sides. If projections of triangles normal on axis look outside of point, the point is inside, if opposite then point is outside, if normal's projection has same direction, then mesh has gaps and axis hit that gap. The third most rare state of point is being on the surface of another mesh, in this case the most close triangle will contain that point.
• When we have mesh with marked point relative to other mesh, we can make desicions triangle-wise. If all points of triangle have same state (all inside for example) and all sides of triangle don't cross any other triangle, then this triangle could be dropped (if it is inside the mesh and we make mesh union, or it outside and we make mesh intersection or substraction) or can be added to the new mesh otherwise (point list of triangle should be reversed to change normal direction in case of substraction, or xor)
• The most complex case if triangles are crossing each other, then we should divide triangle into parts so each of them would have simple borders to other. Each triangle subdivide into up to three triangles by each intersection (we should reserve enought memory).

Operation is heavy indeed. I think it could be nice job for compute shader.

[bugsweeper]

The difference between union, substraction, xor and intersection - which side of each mesh do we choose inside or outside relative to another mesh.
PS: this is very important feature for sculpting or games with desctructions

[bugsweeper]

The other intresting question is: what color should new surface have (only union have no new surface, other operations have cut place, every point of which should have some color or UV coordinate for texture). Should this operations take extra parameter (closure?) which would make such decision? Because for sculpting there is bad decision to leave with new surface (copied from knife surface) the color of knife. For example game Fruit Ninja cuts fruits like watermelon which could be one color (green) outside but very different color inside (red) and this is very new color, not the color from old fruit mesh or katana mesh.

[superdump]

As a note, bevy aims to be able to manage assets on web as well as native. That doesn’t mean that C++ dependencies can’t be used, but bear in mind that they will not be available on web and will likely have to be behind a feature flag as they can’t be considered core functionality given they are not available everywhere. Or something like that. I don’t know if we maintainers have explicitly defined that policy anywhere but the cross-platform support and web asset handling support is clear.

[RCC-CCN]

Just to clarify, my suggestion was not to include a C++ library as an external dependency but as a possibile reference that being C++ would need a substantial effort to transpose code/concepts.

[HackerFoo]

It might be possible to include Manifold as a dependency for WASM builds:

• https://github.com/rafaelbeckel/test-c-rust-wasm
• Align with the "C" ABI rustwasm/wasm-bindgen#3454
• https://github.com/elalish/manifold/blob/master/README.md#wasm

[pca006132]

I have some experience optimizing mesh boolean performance, here are my opinions on some of the points mentioned above:

I think the most simple naive implementation is to build a fn that can take a cuboid and subtract another cuboid.

That is a voxel-based approach, which can work. This approach impose limits on the precision of the input that you can work with, as the number of voxels will quickly blow-up as you increase precision, and the artifacts can be fairly noticeable.
Also, it can be hard to map texture back to the result.

---

Just want to point out there is no need to consume the mesh, taking a reference is just as performant.

No, it's not, because you'll end up doing a lot of cloning, assuming the mesh is first converted to some sort of spatial tree. Unless you mean a mutable reference, which is essentially the same thing, but much harder to manage in this case.

From my experience, cloning the mesh is not that slow comparing to the actual boolean operation. It is practically impossible to do the computation in-place, so you need allocation anyway. The resulting mesh will usually be quite different from the input, so the input buffer usually can't be reused.

Using references will allow you to do some optimization on the csg tree (DAG), enables sharing without expensive cloning, etc. We wrote some general tips here: https://github.com/elalish/manifold/wiki/Performance-Considerations

There may be use cases where you want to modify part of the mesh, but it is hard to make an algorithm that just mutate a part of the mesh that is changed, at least manifold is not able to do that for now (and no one is asking us to do that).

---

This task is not so complex if we divide it to a set of simple tasks (I put here list of naive task list. It based on expectation that mesh is complete and has not gaps. Resulting mesh should also be complete with no gaps):

Probably not that simple, you can also have self-intersection or invalid topology and things will become really complicated, see https://github.com/elalish/manifold/wiki/Manifold-Library

Also, mesh boolean and triangulation are full of corner cases which can be tricky to handle correctly. It took years for manifold to get to the current state, and there are still known issues. Performance is also a difficult subject, it will not be very useful if boolean operation on simple things take hours to complete (e.g. openscad is well-known for this previously when input is complicated).

---

Operation is heavy indeed. I think it could be nice job for compute shader.

This will be very hard because you are often memory-bound (host to device transfers). manifold used to have CUDA backend for some algorithms, but we later ditched it because proper CPU implementation is faster.

[HackerFoo]

There may be use cases where you want to modify part of the mesh, but it is hard to make an algorithm that just mutate a part of the mesh that is changed, at least manifold is not able to do that for now (and no one is asking us to do that).

This would be difficult with a monolithic mesh, but is possible If the mesh/shape is split into smaller components.

[onkoe]

I think Godot has some decent docs on this, plus their source is fairly readable.

I've used these APIs a bit, and while not perfect, they seem more than capable of doing those operations. :)

• docs: Prototyping levels with CSG - CSG tools features
• src: godot/core::math::geometry_2d::Geometry2D

[atlv24]

i don't believe this is at all in scope for bevy, and there's no good justification to do this on bevy's Mesh type. CSG algorithms practically always require additional bookkeeping, which is not something Mesh should have. bevy Mesh should have only one responsibility: rendering.

this is best suited to a third party crate, integrating tightly with bevy Mesh is neither necessary nor desirable. Mesh would only be an output format. Furthermore, this isn't even something that would need to be specific to bevy, CSG is general purpose

[onkoe]

imv, closing this severely limits the usability of Mesh in complex projects. community projects almost always run behind bevy@latest... 😔

if you're available to do so, can you please explain some of the challenges with making this not add overhead to Mesh?

some questions as someone interested in authoring something like this:

• would adding a type that renders a Mesh from combining and caching its inner elements not be in scope of Bevy?
• are there any undocumented performance goals of Mesh in your consideration?
• do you believe Manifold is a good fit for a Bevy plugin?
• is CSG the only non-goal here, or does this being "not planned" also include mesh creation through boolean operations?

[HackerFoo]

The Mesh datatype is designed to hold the input data for a vertex shader. This is quite different from what CSG and other geometry algorithms need. The name is confusing, because a GPU mesh is typically a list of triangles along with attributes, which don't hold any connectivity beyond the three vertices of each triangle, while a mesh used in geometry contains a lot more information about how vertices are connected. Furthermore, geometry meshes do not need to be tessellated into triangles. So the only thing they may have in common is the positions of the vertices.

Maybe mesh shaders will allow conversion of geometry meshes to GPU meshes on a GPU, or even a compute shader, but this would be outside the scope of Bevy, other than making it possible to do this.

[alice-i-cecile]

I would be open to "mesh modification and CSG" in the future, almost certainly via upstreaming an established community crate. I think it's an important problem domain, and spent months seriously pursuing a commercial product with Bevy in this space.

But as HackerFoo says, Bevy's Mesh type isn't designed for that use, and should be optimized for rendering.

That said, I suspect that said mesh geometry crate is likely going to be easier to maintain and more broadly useful if it's (mostly) not Bevy specific. This is a "data structures with methods" sort of domain: as long as you can convert to and from Bevy's rendering representation you should be good.

[atlv24]

Bevy will always be able to take a set of vertices and indices and render them as a mesh. A CSG library can produce a set of vertices and indices as output from its internal data structures. If you wanna be fancy there can be a bevy_csg layer which takes a dep on both of these and provides the glue code layer, but it would likely be 5-10 lines of actual logic at most.

[onkoe]

Got it! So, to be clear, would a crate that:

• is MIT-licensed, and
• has a general algorithm and caching type, but provides enough info through methods to solve for Bevy's meshes

be considered for inclusion within Bevy?

In particular, does the Manifold library meet these requirements? I'm interested in wrapping its build process and API into a portable Rust crate, so it's hard to know where to draw the line.

(also, thank you for the detailed replies - yall painted an informative picture!)

[alice-i-cecile]

MIT + Apache licensing for upstreaming :)

In this specific case, we'd want the solution to be Bevy-optional. That could either be a shim crate (like bevy_rapier) or a 'bevy' feature, like bevy_reflect. The former (and external ownership of the crate) is likely going to be preferred to reduce maintenance burden and improve ecosystem collaboration in this case.

Bevy should support CSG operations and mesh modifications at some point, but we don't always need to go full NIH.

I have no specific knowledge of Manifold itself, but something well-made, well-maintained and ideally pure Rust is something we would seriously consider building a dedicated integration for. I would say there's almost no chance we choose to maintain a C++ integration layer though: we don't have the expertise and it substantially complicates the user and contributor experience.