MONAI translation to BioImage format

[IvanHCenalmor]

Hi,
I have been working with the script to translate the MONAI models to BioImage format. I already have some of them working with pre and post processing:

• BraTS MRI segmentation
• Breast density classification
• Endoscopic inbody classification
• Pathology tumor detection

The pre and post processing functions are still for numpy array objects, some of them have already been translated to Xarray format and the rest of them should be easy as they have similar formats.

I have printed a some graphs to visualize all the pre and post processing functions that MONAI has and in which of their models are they used (white means that they are using it and black that they don't):

• Preprocessing:
• Postprocessing:

There are many functions that are only used by a single model, for example valve_landmarks, which has both single-use pre-repocessing and post-processing functions.

I wanted to discuss a bit about what should be done, because the translation of some of the functions is really complex and sometimes is only applied to Pytorch tensors that are the ones used by MONAI models. I will divide it in different points that I think that are important:

• First, about having the dependency of MONAI in BioImageIO. In order to read their configuration files (yaml or json) and load the model they are using, I am using a class from MONAI's library that is ConfigParser from monai.bundle. I think that without this dependency it will be hard to automatically load the model. Maybe the dependency from MONAI is needed if we want to take their models, if not the solution would be to manually obtain all their models and save them. What do you think is the best option?
• For the images, it has been hard but I think that except from renalStructures_UNEST_segmentation_v0.2.2 I have an input image for the rest of the models.
• The pre-processing and post-processing functions. There are some of them that are easy to translate or that are already in BioImageIO, but there are others that are really specific and that are really complex. Also, the fact that they work with Pytoch tensors and we work with Xarray is making the things more complex, because many of the functions like reshaping are not directly implemented in numpy or Xarray. How bad would it be to adopt the MONAI preprocessing functions directly? At the end the model needs to receive a Pytorch tensor as input. Is not an option to save the metadata from the input image, preprocess the images with MONAI's functions and then reuse the metadata that was saved? If we still want to implement our own pre and post processing functions, what should we do with those complex functions? Should I create a pull request with the functions that I already have or should I wait until we decide what we should do?
• Also, some of the models predict a dictionary, which is a problem for the actual processing functions that receive an Xarray.

Maybe the option of having MONAI has a dependency is not that bad option. The process is more complex and not that modular as I thought at the beginning, they have very specific functions and configurations for many models.

What do you think?

[FynnBe]

Awesome overview of the pre- and postprocessing in use! This makes it clear that we can go a long way with a few standardized pre-/postprocessing operations!

First, about having the dependency of MONAI in BioImageIO. In order to read their configuration files (yaml or json) and load the model they are using, I am using a class from MONAI's library that is ConfigParser from monai.bundle. I think that without this dependency it will be hard to automatically load the model. Maybe the dependency from MONAI is needed if we want to take their models, if not the solution would be to manually obtain all their models and save them. What do you think is the best option?

Those models for wich we can (in the foreseeable future) use bioimageio.core pre- and postprocessing we should extract the required data and upload each model to their own zenodo record -- like any bioimageio model.

Those that definitely need the monai library dependency we'll tackle afterwards. (there are two avenues: using the dependencies field in the spec or going via bioimageio.workflows) Let's learn from the simple cases first before thinking about these more complex models further.

For the images, it has been hard but I think that except from renalStructures_UNEST_segmentation_v0.2.2 I have an input image for the rest of the models.

Awesome! let's ignore this one just for now

The pre-processing and post-processing functions. There are some of them that are easy to translate or that are already in BioImageIO, but there are others that are really specific and that are really complex....

Don't translate any processing functions that give you a headache! Do some easy ones and let's get the first MONAI model in the model zoo through zenodo as a "standard" bioimagieo model. We'll see how far we get. There will definitely be a point for the more complex models where we need the monai library. Where exactly this threshold falls is not that important.

Also, some of the models predict a dictionary, which is a problem for the actual processing functions that receive an Xarray.

If it's a simple dictionary with string keys (e.g. a dictionary with some named tensors) then this translates nicely to multiple outputs of a bioimageio model... If not, ignore for now, maybe write it on a list to keep an overview of these cases.

Maybe the option of having MONAI has a dependency is not that bad option. The process is more complex and not that modular as I thought at the beginning, they have very specific functions and configurations for many models.

If the simplest model is already too complex, let's take a look together, but ultimately that totally is a valid outcome and we'll simply have to look into the general case next.

So, don't focus on solving the translation for all models at the same time. You gave us a good overview, now let's look at the models from simplest to most complex and see when we hit the complexity threshold.