Converted ONNX model accuracy drops about 10-15% when exporting pytorch model

[LuisGF93]

Hello,

I've trained an efficientnet v2 model using this repository and then I've converted it to ONNX format.
Then I have run inference on my pth checkpoint over a set of images to see its accuracy and I tried to run inference over the same set of images with my ONNX converted model. There are no errors in the log, apparently everything goes fine with the conversion process and also with the inference processes, but the accuracy on my converted model is between a 10% and a 15% worse than the original.

Do you know where could be the problem?

[rwightman]

@LuisGF93 can you specifiy which models? if it's the tf_ variants it's likely the same padding hack that's added to make pytorch models compat with the tensorflow padding. The few models I've trained myself without the tf_ prefix shouldn't have that problem. If they do it's likely an onnx / pytorch compat issue as those crop up from time to time.

[LuisGF93]

Thank you for your reply.

The architecture I trained is efficientnetv2_rw_t, do you how could I resolve that issue?

[rwightman]

@LuisGF93 hmm, haven't checked onnx in a bit, but it used to work for these models. However, different versions of PyTorch / ONNX have a tendency to break things... I'd need all versions used and a link to or gist of the code used to export and validate.

[LuisGF93]

The code I use to convert is the following:

import torch
from timm.models import create_model
import os

model_path ='/path/to/pth.tar'
model_export = '/path/to/onnx.onnx'
model_arch = 'efficientnetv2_rw_t'
input_shape = [1, 3, 224, 224]
num_classes = 70

# create model
base_model = create_model(
    model_arch,
    num_classes=num_classes,
    in_chans=3,
    checkpoint_path=model_path)

model = torch.nn.Sequential(
    base_model,
    torch.nn.Softmax(dim=1)
)

model.eval()

dummy_input = torch.randn(1, 3, 224, 224, requires_grad=True)

torch.onnx.export(model,
                  dummy_input,
                  model_export,
                  verbose=False,
                  export_params=True,
                  opset_version=10,  # the ONNX version to export the model to
                  do_constant_folding=True  # whether to execute constant
                  )

and the script to run the onnx inference is like this:

import os
import numpy as np
import torchvision
from PIL import Image
import onnxruntime

labels_file = "/path/to/labels.txt"
onnx_path = "/path/to/onnx.onnx"
images_folder = "/path/to/test_images"
input_size = (224, 224)

with open(labels_file, 'r') as label_file:
    line = label_file.readline()
labels = line.split(';')

for directory in sorted(os.listdir(images_folder)):
        folders_list.append(directory)

# Onnx config
sess = onnxruntime.InferenceSession(onnx_path)

print('--- Running Inference ---')
for directory in folders_list:
        for img in os.listdir(images_folder + directory):
            if img.endswith(".jpg"):
                img_dict = dict()
                max_index = -1
                max = float('-inf')
                # Transformations for the input data
                transforms = torchvision.transforms.Compose([
                    torchvision.transforms.Resize([224, 224]),
                    torchvision.transforms.ToTensor(),
                    torchvision.transforms.Normalize(mean=[0.485, 0.456, 0.406], 
                    std=[0.229, 0.224, 0.225])
                    ])
                input_img = Image.open(/path/to/image).convert('RGB')
                # Transformation
                input_data = transforms(input_img)
                input_data.unsqueeze_(0)
                image = np.array(input_data.cpu().numpy(), dtype=np.float32)
                # Inference
                input_name = sess.get_inputs()[0].name
                label_name = sess.get_outputs()[0].name
                result = sess.run(None, {input_name: image})
                
                for i in range(0, len(result[0][0])):       
                    if(result[0][0][i] > max):    
                        max = result[0][0][i]
                        max_index = i

                img_dict['File'] = /path/to/image
                img_dict["Prediction"] = labels[max_index]
                img_dict["Real"] = directory
                img_dict['Correct'] = 0
                if img_dict["Real"] == img_dict["Prediction"]:
                    img_dict['Correct'] = 1
                img_dict_list.append(img_dict)

# Export a csv file after inference

The packages I'm using are these:

timm==0.5.4
torch==1.10.1
torchaudio==0.10.1+cpu
torchvision==0.11.2
onnxruntime==1.10.0

with python 3.6

[rwightman]

@LuisGF93 hmm, I have some onnx export and validation code in another repo, although it's likely out of date with current onnx / pytorch but may be some differences to look at, in the past it was easy to swap the gefnet factor for the timm one...

• https://github.com/rwightman/gen-efficientnet-pytorch/blob/master/onnx_validate.py
• https://github.com/rwightman/gen-efficientnet-pytorch/blob/master/onnx_export.py

In yours you are using 224 which is not the res those models are eval with, you are also using bilinear for resize, the eval # are at 288x288 for that model, you should resize to 288 (no tuple, so that it does resize shortest edge), use bicubic interpolation, and center crop to square. Though that should be a 2-3% difference, not 10-15%.