Getting CPP TensorRT example to work/garbage output

[charlesrwest]

Hello!

I'm trying to get PyTorch-ONNX-TensorRT-CPP working on Manjaro Linux (cuda 11.0, opencv 4.4.0 with cuda support, tensorrt 7.2, pytorch 1.6). There have been a few problems and I'm not sure what to try next:

1. It compiles fine, but some of the used features are depreciated:
   [ 50%] Building CXX object CMakeFiles/trt_sample.dir/trt_sample.cpp.o /home/charlesrwest/cpp/libraries/learnopencv/PyTorch-ONNX-TensorRT-CPP/trt_sample.cpp: In function ‘void parseOnnxModel(const string&, TRTUniquePtr<nvinfer1::ICudaEngine>&, TRTUniquePtr<nvinfer1::IExecutionContext>&)’: /home/charlesrwest/cpp/libraries/learnopencv/PyTorch-ONNX-TensorRT-CPP/trt_sample.cpp:144:79: warning: ‘virtual nvinfer1::INetworkDefinition* nvinfer1::IBuilder::createNetwork()’ is deprecated [-Wdeprecated-declarations] 144 | TRTUniquePtr<nvinfer1::INetworkDefinition> network{builder->createNetwork()}; | ^ In file included from /home/charlesrwest/cpp/libraries/learnopencv/PyTorch-ONNX-TensorRT-CPP/trt_sample.cpp:3: /usr/include/NvInfer.h:6954:58: note: declared here 6954 | TRT_DEPRECATED virtual nvinfer1::INetworkDefinition* createNetwork() TRTNOEXCEPT = 0; | ^~~~~~~~~~~~~ [100%] Linking CXX executable trt_sample [100%] Built target trt_sample

2. Attempting to run it with the given onnx file fails

build]$ ./trt_sample ../resnet50.onnx ../turkish_coffee.jpg [libprotobuf ERROR /home/charlesrwest/cpp/libraries/tensorrt/src/build/third_party.protobuf/src/third_party.protobuf/src/google/protobuf/text_format.cc:309] Error parsing text-format onnx2trt_onnx.ModelProto: 1:9: Message type "onnx2trt_onnx.ModelProto" has no field named "version". Failed to parse ONNX model from file../resnet50.onnx Network must have at least one output Network validation failed. Segmentation fault (core dumped)
3. Updating the model succeeds:

python3 ./pytorch_model.py class: cup , confidence: 92.43077850341797 %, index: 968 class: espresso , confidence: 6.13804292678833 %, index: 967 class: coffee mug , confidence: 0.7285550832748413 %, index: 504 Model was successfully converted to ONNX format. It was saved to resnet50.onnx
4. But running on the updated model fails

`./trt_sample ../resnet50.onnx ../turkish_coffee.jpg

Input filename: ../resnet50.onnx
ONNX IR version: 0.0.6
Opset version: 9
Producer name: pytorch
Producer version: 1.6
Domain:
Model version: 0
Doc string:

ERROR: /home/charlesrwest/cpp/libraries/tensorrt/src/TensorRT/parsers/onnx/ModelImporter.cpp:500 In function importModel:
[4] Assertion failed: !_importer_ctx.network()->hasImplicitBatchDimension() && "This version of the ONNX parser only supports TensorRT INetworkDefinitions with an explicit batch dimension. Please ensure the network was created using the EXPLICIT_BATCH NetworkDefinitionCreationFlag."
ERROR: could not parse the model.
Segmentation fault (core dumped)
5. This can be fixed by adding a flag when building the engine:
TRTUniquePtrnvinfer1::INetworkDefinition network{builder->createNetwork()};
->
const auto explicitBatch = 1U << static_cast<uint32_t>(nvinfer1::NetworkDefinitionCreationFlag::kEXPLICIT_BATCH);
TRTUniquePtrnvinfer1::INetworkDefinition network{builder->createNetworkV2(explicitBatch)};
`
6. The teensorrt netwrk then runs, but gives a garbage output

`
./build/trt_sample ./resnet50.onnx ./turkish_coffee.jpg

Input filename: ./resnet50.onnx
ONNX IR version: 0.0.6
Opset version: 9
Producer name: pytorch
Producer version: 1.6
Domain:
Model version: 0
Doc string:

class: web site, website, internet site, site | confidence: 5.84856% | index: 916
class: cleaver, meat cleaver, chopper | confidence: 3.78223% | index: 499
class: nematode, nematode worm, roundworm | confidence: 3.32261% | index: 111
class: envelope | confidence: 3.00015% | index: 549
class: hatchet | confidence: 1.46123% | index: 596
class: nail | confidence: 1.37108% | index: 677
class: whistle | confidence: 1.23372% | index: 902
class: can opener, tin opener | confidence: 1.171% | index: 473
class: hook, claw | confidence: 1.13449% | index: 600
class: guillotine | confidence: 0.982188% | index: 583
class: theater curtain, theatre curtain | confidence: 0.93405% | index: 854
class: letter opener, paper knife, paperknife | confidence: 0.930356% | index: 623
class: panpipe, pandean pipe, syrinx | confidence: 0.855095% | index: 699
class: revolver, six-gun, six-shooter | confidence: 0.823099% | index: 763
class: safety pin | confidence: 0.822059% | index: 772
class: plunger, plumber's helper | confidence: 0.785783% | index: 731
class: scabbard | confidence: 0.769268% | index: 777
class: hair slide | confidence: 0.726561% | index: 584
class: bulletproof vest | confidence: 0.715805% | index: 465
class: table lamp | confidence: 0.715662% | index: 846
class: book jacket, dust cover, dust jacket, dust wrapper | confidence: 0.667637% | index: 921
class: assault rifle, assault gun | confidence: 0.663674% | index: 413
class: stopwatch, stop watch | confidence: 0.662865% | index: 826
class: abaya | confidence: 0.655316% | index: 399
class: binder, ring-binder | confidence: 0.63594% | index: 446
class: chain | confidence: 0.635571% | index: 488
class: dishwasher, dish washer, dishwashing machine | confidence: 0.618163% | index: 534
class: rule, ruler | confidence: 0.606973% | index: 769
class: fire screen, fireguard | confidence: 0.605063% | index: 556
class: lipstick, lip rouge | confidence: 0.594731% | index: 629
class: screw | confidence: 0.585789% | index: 783
class: muzzle | confidence: 0.576213% | index: 676
class: mousetrap | confidence: 0.57476% | index: 674
class: ocarina, sweet potato | confidence: 0.572207% | index: 684
class: magnetic compass | confidence: 0.535981% | index: 635
class: hammer | confidence: 0.524395% | index: 587
class: maraca | confidence: 0.512932% | index: 641
`

[zacharyfrederick]

Did you ever get this figured out? I am stuck with the same issue.

[ducdung98tb]

I have the same problem. Have you solved it yet?

[ducdung98tb]

auto input_width = dims.d[2];
auto input_height = dims.d[3];
auto channels = dims.d[1];

A change like above should solve the problem.

[seongkyun]

To short,

1. update the resnet50.onnx file by running 'python3 ./pytorch_model.py' in the terminal.
2. change the function named "parseOnnxModel"
   from
   TRTUniquePtr<nvinfer1::INetworkDefinition> network{builder->createNetwork()};
   to
   const auto explicitBatch = 1U << static_cast<uint32_t>(nvinfer1::NetworkDefinitionCreationFlag::kEXPLICIT_BATCH); TRTUniquePtr<nvinfer1::INetworkDefinition> network{builder->createNetworkV2(explicitBatch)};
3. change the function named "preprocessImage"
   from
   auto input_width = dims.d[2]; auto input_height = dims.d[1]; auto channels = dims.d[0];
   to
   auto input_width = dims.d[2]; auto input_height = dims.d[3]; auto channels = dims.d[1];
4. make it and run
   $ cmake -DOpenCV_DIR=/workspace/shared_data/opencv/opencv-4.5.1 -DTensorRT_DIR=/workspace/tensorrt .. $ make -j8 $ ./trt_sample ../resnet50.onnx ../turkish_coffee.jpg
   checked it on Ubuntu18.04 LTS, CUDA11.1, cuDNN8.1, OpenCV4.5.1 with cuda in nvidia-docker tensorrt20.10 image.

[Shahd-A-Mahmoud]

@seongkyun @charlesrwest @ducdung98tb @zacharyfrederick
Note the true dimension is auto input_width = dims.d[3]; auto input_height = dims.d[2]; auto channels = dims.d[1];
your solution swap the width by the height and this brings noises in the output

[Shahd-A-Mahmoud]

@seongkyun @charlesrwest @ducdung98tb @zacharyfrederick Note the true dimension is auto input_width = dims.d[3]; auto input_height = dims.d[2]; auto channels = dims.d[1]; your solution swap the width by the height and this brings noises in the output

I have successfully solved the issue by doing that