OpenCV Implementation

[felixcantet]

Open CV Integration into Mixture

Motivation

OpenCV (Open Computer Vision Library) is a massive Library composed of lot of optimized state of the art Computer Vision algorithm, from Image Segmentation to basic image processing...

It makes sense for Mixture to have a lot of the algorithms implemented in OpenCV available in the graph.

OpenCV if mostly CPU based, so we can take advantage of it for some algorithm that are more suited for CPU processing.

Implementation Details

OpenCvSharp Bindings

I use OpenCvSharp as Binding for native OpenCV Dll.

Crossplatform

Currently, only the windows binding is contained in the Draft. I have to make sure Linux and Mac are included too.

Dlls

I have use NuGetForUnity package to install OpenCvSharp Nuget package with all the dependencies in Unity. I didn't configure it, so all the dlls are install in the Assets/Packages folder (default behaviour of the NuGetForUnity package). I have to make sure the Dll's are placed in the Mixture package folder.
NuGetForUnity also download lot of configuration files, notice, etc... I have to clean it up so we only have necessary files in the repository.

Data Conversion

Cv::Mat

OpenCv is using it's own texture type : Cv::Mat that support 1 to 4 channels on any number of dimensions with 8 to 32bits per channel. I've created a MixtureOpenCvUtils static class that simplify the conversion from Texture to Mat and vice versa, choosing the right format for each and copy data in an optimized way from one to another

RGBA vs BGRA

OpenCV default color format is BGRA (don't ask me why...), so we have to make sure colors are converted correctly before converting data from Unity to OpenCV and vice-versa

Math Types

OpenCV has it's own math types :

• Vec{size}{type abreviation} (ex : Vec3f for float Vector3, Vec4b for Vector4 of byte)...
• Size
• Rect
• Scalar
  It also extensively use double type

How to make the user experience seamless across this duplicated datatypes? Does NGP support custom casting ? (as exemple, if the input is a Vec3f, allow casting from Vector3 to Vec3f ? Because we can't edit Unity source code neither OpenCV source code, adding custom cast operator is not possible for these duplicated types

Algorithms

Here is a list of the algorithm we want to integrated using OpenCV :

Flood Fill node set

Similar toolset as Substance Designer Flood Fill.
This nodes will allow to detect each connected shape and apply independent processing to each of them.
In substance Designer, there are many nodes related to this Flood Fill :

• Flood Fill To Gradient
• Flood Fill to Random Color
• Flood Fill to BBox Size
• Flood Fill to Color, etc...

Simple Flood FIll

Given an image, a seed point and a color, color all connected pixels to the seed to the given color. A connected pixel is link to the seed by a continuity of similar color

Other needs to be added here

Open Questions

Do we want low level nodes for OpenCV ?

The low level nodes of OpenCV would allow to use Mixture as a Visual Scripting Engine for OpenCV.
In the other hand, we could use OpenCV only to implement high level nodes and make the use of the library abstract (Flood Fill similar to Substance Designer, or other...)

[alelievr]

I see quite a few files outside of the com.alelievr.mixture package folder, these will not be installed if the package is installed through the package manager of Unity.
I think it's important to investigate if it's possible to:

• Add NugetForUnity inside Mixture + handle if it's already installed in the project somehow.
• Automatically install dependencies from Nuget in the project
• Add an asmdef for OpenCV nodes so that when OpenCV is not install it still compiles

[felixcantet]

Thanks for the feedbacks. Yes, I was pointing the file location issue in the draft.
I was thinking of just using locally NugetForUnity to get the files on my side and commit the dll's manually. But maybe adding NugetForUnity as a package dependency and creating a small editor tool to install OpenCV bindings through the NugetForUnity API could be a solution.