ASP.NET vs ML.Net ObjectDetection

[schalke046]

System Information (please complete the following information):

• OS & Version: Docker linux
• ML.NET Version: latest
• .NET Version: 8.0

Describe the bug
Hi all,

I'm currently having an issue with object detection in the docker container by using TorchSharp-Cpu. The first object detection is going fine by having around 2 GB in use, but after more like 100 the RAM usage is going up to 4GB and more.
And the worst part of it is that this is running out of memory after a while.
Looking at the process after the usage was done, there was no decrease in memory usage happening.

The model is created by images up to 9MB PNGs but resized in the model to only 250 by 250.

I was already multiple times checking the using statements and all IDisposable objects.

To Reproduce
Steps to reproduce the behavior:

1. Set Up ASP.Web API
2. Add an object detection Model => train it etc.
3. Add a controller to use the model
4. Provide an upload function with IFormFile
5. Add using on opening the IFormFileStream
6. Add using for the ModelInput
7. Add a result around the prediction, store the result of precision in that, and close or make a using the output image.
8. Make sure all using's are there and all streams are going to be closed.
9. Optional (Add a lock statement around prediction to make it anytime single-threaded)

Expected behavior
RAM usage for single-threaded usage should go not over 2.5 GB or come down after usage. Dot. Memory on Windows was only showing that there is a lot of unmanaged Memory in use (streams).

Screenshots, Code, Sample Projects

[LittleLittleCloud]

@schalke046 Can you provide a minimum reproducable example? With just text description it's hard to figure out where goes wrong on our end

[schalke046]

Hi @LittleLittleCloud,

for sure. I uploaded a dice detector example. Please also check that all things are disposed, because I made a copy of the original code.

OneDrive link with model and example images
https://1drv.ms/u/s!AiNaHhnlJdwuh99lZaBCid0EuHYDdQ

Thanks
schalke046

[LittleLittleCloud]

@schalke046 Is that only happen in docker scenario? And which tool are you using to produce the screenshot you hsare in the issue

[schalke046]

@LittleLittleCloud, I didn't test that code on a normal run. But I also noticed it on normal Windows debugging.

[schalke046]

With that code, it seems to be stocked on my laptop at around 1.5GB of RAM, but Docker linux use around 3.1 GB RAM and more.

[LittleLittleCloud]

@schalke046
I'm now running your sample on my windows desktop and the memory consumption is 5.8GB at the current moment.

I'm going to keep the program on for a bit longer to see how much memory it's going to take in the end.

In the meantime, I also take a look at the object detection trainer and it seems that the image tensor is not disposed.

Could it be what causing the memory consumption keep growing issue @michaelgsharp ?

machinelearning/src/Microsoft.ML.TorchSharp/AutoFormerV2/ObjectDetectionTrainer.cs

Line 968 in fa8c822

var imageTensor = PrepInputTensors(ref image, getImage);

[LittleLittleCloud]

@schalke046

I also notice that you are using mlnet 3.0.0, maybe consider upgrading to the latest preview as if I remember correctly there were a few fix for memory consumption issue for torchsharp model. So maybe upgrade to latest preview and see if memory pressure get reduced

	  <PackageReference Include="Microsoft.ML" Version="4.0.0-preview.24378.1" />
	  <PackageReference Include="Microsoft.ML.ImageAnalytics" Version="4.0.0-preview.24378.1" />
	  <PackageReference Include="Microsoft.ML.TorchSharp" Version="0.22.0-preview.24378.1" />
	  <PackageReference Include="TorchSharp-cpu" Version="0.102.7" />

[schalke046]

Sure I'll have a try

[schalke046]

Hi @LittleLittleCloud,
It's not so much better, but it has less memory usage. After a quick break (~ 1 minute) on uploading it's increasing and no cleanup was happening.

Also after 3 minutes is no cleanup happening. Also, docker was not responding anymore... I'm expecting docker was out of memory.

[schalke046]

@LittleLittleCloud with the latest preview version it's working pretty well on Windows.

With three threads on the testing application and a lock on the prediction engine, it's working with 2GB of RAM. Only Docker / Linux have these issues. Updated code https://1drv.ms/f/s!AiNaHhnlJdwuh99kZNUZRp-rQ2qh3Q?e=8bWGly