"Segmentation fault (core dumped)" for new JetPack 6.1 install on Orin Nano when running demo

[verderog]

I'm experiencing a segmentation fault trying to run the segnet demo with Jetpack 6.1 on the Orin Nano.

Steps to replicate:

1. Download and write the JP 6.1 image from https://developer.nvidia.com/downloads/embedded/l4t/r36_release_v4.0/jp61-orin-nano-sd-card-image.zip to micro SD card.

2. Boot new image. Perform initial user setup, config, etc.

3. Follow jetson-inference instructions here: https://github.com/dusty-nv/jetson-inference/blob/master/docs/building-repo-2.md. Install PyTorch when prompted.

4. cd data/images in this repo.

5. segnet.py airplane_0.jpg airplane_0_output.jpg. I've also ran segnet with the same segmentation fault.

6. Encounter segmentation fault. See below for log.

segNet -- loading segmentation network model from:
       -- prototxt:   
       -- model:      networks/FCN-ResNet18-Pascal-VOC-320x320/fcn_resnet18.onnx
       -- labels:     networks/FCN-ResNet18-Pascal-VOC-320x320/classes.txt
       -- colors:     networks/FCN-ResNet18-Pascal-VOC-320x320/colors.txt
       -- input_blob  'input_0'
       -- output_blob 'output_0'
       -- batch_size  1

[TRT]    TensorRT version 10.3.0
[TRT]    loading NVIDIA plugins...
[TRT]    Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
[TRT]    Registered plugin creator - ::BatchedNMS_TRT version 1
[TRT]    Registered plugin creator - ::BatchTilePlugin_TRT version 1
[TRT]    Registered plugin creator - ::Clip_TRT version 1
[TRT]    Registered plugin creator - ::CoordConvAC version 1
[TRT]    Registered plugin creator - ::CropAndResizeDynamic version 1
[TRT]    Registered plugin creator - ::CropAndResize version 1
[TRT]    Registered plugin creator - ::DecodeBbox3DPlugin version 1
[TRT]    Registered plugin creator - ::DetectionLayer_TRT version 1
[TRT]    Registered plugin creator - ::EfficientNMS_Explicit_TF_TRT version 1
[TRT]    Registered plugin creator - ::EfficientNMS_Implicit_TF_TRT version 1
[TRT]    Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1
[TRT]    Registered plugin creator - ::EfficientNMS_TRT version 1
[TRT]    Could not register plugin creator -  ::FlattenConcat_TRT version 1
[TRT]    Registered plugin creator - ::GenerateDetection_TRT version 1
[TRT]    Registered plugin creator - ::GridAnchor_TRT version 1
[TRT]    Registered plugin creator - ::GridAnchorRect_TRT version 1
[TRT]    Registered plugin creator - ::InstanceNormalization_TRT version 1
[TRT]    Registered plugin creator - ::InstanceNormalization_TRT version 2
[TRT]    Registered plugin creator - ::InstanceNormalization_TRT version 3
[TRT]    Registered plugin creator - ::LReLU_TRT version 1
[TRT]    Registered plugin creator - ::ModulatedDeformConv2d version 1
[TRT]    Registered plugin creator - ::MultilevelCropAndResize_TRT version 1
[TRT]    Registered plugin creator - ::MultilevelProposeROI_TRT version 1
[TRT]    Registered plugin creator - ::MultiscaleDeformableAttnPlugin_TRT version 1
[TRT]    Registered plugin creator - ::NMSDynamic_TRT version 1
[TRT]    Registered plugin creator - ::NMS_TRT version 1
[TRT]    Registered plugin creator - ::Normalize_TRT version 1
[TRT]    Registered plugin creator - ::PillarScatterPlugin version 1
[TRT]    Registered plugin creator - ::PriorBox_TRT version 1
[TRT]    Registered plugin creator - ::ProposalDynamic version 1
[TRT]    Registered plugin creator - ::ProposalLayer_TRT version 1
[TRT]    Registered plugin creator - ::Proposal version 1
[TRT]    Registered plugin creator - ::PyramidROIAlign_TRT version 1
[TRT]    Registered plugin creator - ::Region_TRT version 1
[TRT]    Registered plugin creator - ::Reorg_TRT version 2
[TRT]    Registered plugin creator - ::Reorg_TRT version 1
[TRT]    Registered plugin creator - ::ResizeNearest_TRT version 1
[TRT]    Registered plugin creator - ::ROIAlign_TRT version 1
[TRT]    Registered plugin creator - ::ROIAlign_TRT version 2
[TRT]    Registered plugin creator - ::RPROI_TRT version 1
[TRT]    Registered plugin creator - ::ScatterElements version 2
[TRT]    Registered plugin creator - ::ScatterElements version 1
[TRT]    Registered plugin creator - ::ScatterND version 1
[TRT]    Registered plugin creator - ::SpecialSlice_TRT version 1
[TRT]    Registered plugin creator - ::Split version 1
[TRT]    Registered plugin creator - ::VoxelGeneratorPlugin version 1
[TRT]    completed loading NVIDIA plugins.
[TRT]    detected model format - ONNX  (extension '.onnx')
[TRT]    desired precision specified for GPU: FASTEST
[TRT]    requested fasted precision for device GPU without providing valid calibrator, disabling INT8
[TRT]    [MemUsageChange] Init CUDA: CPU +13, GPU +0, now: CPU 41, GPU 2406 (MiB)
[TRT]    Trying to load shared library libnvinfer_builder_resource.so.10.3.0
[TRT]    Loaded shared library libnvinfer_builder_resource.so.10.3.0
[TRT]    [MemUsageChange] Init builder kernel library: CPU +927, GPU +687, now: CPU 1011, GPU 3116 (MiB)
[TRT]    CUDA lazy loading is enabled.
[TRT]    native precisions detected for GPU:  FP32, FP16, INT8
[TRT]    selecting fastest native precision for GPU:  FP16
[TRT]    found engine cache file /usr/local/bin/networks/FCN-ResNet18-Pascal-VOC-320x320/fcn_resnet18.onnx.1.1.100300.GPU.FP16.engine
[TRT]    found model checksum /usr/local/bin/networks/FCN-ResNet18-Pascal-VOC-320x320/fcn_resnet18.onnx.sha256sum
[TRT]    echo "$(cat /usr/local/bin/networks/FCN-ResNet18-Pascal-VOC-320x320/fcn_resnet18.onnx.sha256sum) /usr/local/bin/networks/FCN-ResNet18-Pascal-VOC-320x320/fcn_resnet18.onnx" | sha256sum --check --status
[TRT]    model matched checksum /usr/local/bin/networks/FCN-ResNet18-Pascal-VOC-320x320/fcn_resnet18.onnx.sha256sum
[TRT]    loading network plan from engine cache... /usr/local/bin/networks/FCN-ResNet18-Pascal-VOC-320x320/fcn_resnet18.onnx.1.1.100300.GPU.FP16.engine
[TRT]    device GPU, loaded /usr/local/bin/networks/FCN-ResNet18-Pascal-VOC-320x320/fcn_resnet18.onnx
[TRT]    Loaded engine size: 22 MiB
[TRT]    Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
[TRT]    Deserialization required 13320 microseconds.
[TRT]    Total per-runner device persistent memory is 0
[TRT]    Total per-runner host persistent memory is 128224
[TRT]    Allocated device scratch memory of size 4915200
[TRT]    - Runner scratch: 4915200 bytes
[TRT]    [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +5, now: CPU 0, GPU 27 (MiB)
[TRT]    CUDA lazy loading is enabled.
[TRT]    
[TRT]    CUDA engine context initialized on device GPU:
[TRT]       -- layers       25
[TRT]       -- maxBatchSize 1
[TRT]       -- deviceMemory 4915200
[TRT]       -- bindings     2
[TRT]       binding 0
                -- index   0
                -- name    'input_0'
                -- type    Row major linear FP32 format (kLINEAR)
                -- in/out  INPUT
                -- device  DEVICE
                -- # dims  4
                -- dim #0  1
                -- dim #1  3
                -- dim #2  320
                -- dim #3  320
[TRT]       binding 1
                -- index   1
                -- name    'output_0'
                -- type    Row major linear FP32 format (kLINEAR)
                -- in/out  OUTPUT
                -- device  DEVICE
                -- # dims  4
                -- dim #0  1
                -- dim #1  21
                -- dim #2  10
                -- dim #3  10
[TRT]    
[TRT]    binding to input 0 input_0  binding index:  0
[TRT]    binding to input 0 input_0  dims (b=1 c=3 h=320 w=320) size=1228800
[TRT]    binding to output 0 output_0  binding index:  1
[TRT]    binding to output 0 output_0  dims (b=1 c=1 h=21 w=10) size=8400
Segmentation fault (core dumped)

[rpstandard]

same issue.. moved to using Jetpack 6.0 and that worked for now.

[Granluke]

I am having the same problem with detectnet

[dusty-nv]

@Granluke sorry yes, TRT10 removed deprecated support for caffemodels and TensorFlow UFF models, which this tutorial used a lot having been started in that era. This repo also is based in C++ being created almost 8 years ago, that was more normal for the time and before there was Python bindings for TensorRT or good PyTorch support in ONNX/ect.

Still trying to find a solution for this on JP 6.1 without migrating all the models, because they aren't VITs and not the focus moving forward. For now would use JP 6.0 if you require the repo, sorry for the inconvenience. If you get the jetson-inference:r36.3 docker image running on JP 6.1 / r36.4, let me know 👍

[Granluke]

@dusty-nv No worries, really appreciate everything you are doing! However I am encountering this issue with .onnx models, which I thought TRT10 still supports.

[dusty-nv]

Yep, it does, that's now the primary way to import into TRT without using it's model/layer API directly. However there were other deprecations/removals from TRT10 that changed other inferencing APIs, and making segfault. Apologize that have not had the time to go back and figure it out.

For newer stuff using TRT, I mostly use torch2trt for approaches like clip_trt where it just drops-in to existing PyTorch code replacing the vanilla model. Ultralytics also has native TRT support (with INT8) now. With the more rapid pace of model development now, it is easier keeping that stuff updated because it is in-tree with the models.

EDIT: I still use jetson-utils on jetpack 6.1, that works fine and has persisted in being useful

[smbunn]

@Granluke sorry yes, TRT10 removed deprecated support for caffemodels and TensorFlow UFF models, which this tutorial used a lot having been started in that era. This repo also is based in C++ being created almost 8 years ago, that was more normal for the time and before there was Python bindings for TensorRT or good PyTorch support in ONNX/ect.

Still trying to find a solution for this on JP 6.1 without migrating all the models, because they aren't VITs and not the focus moving forward. For now would use JP 6.0 if you require the repo, sorry for the inconvenience. If you get the jetson-inference:r36.3 docker image running on JP 6.1 / r36.4, let me know 👍

I have jeston-inference:r36.3.0 running under jetpack 6.1 R36.4 and can confirm almost all of the mocels no longer run. Detectnet does, but most others fail. Most common error is simply that the CSI cameras do not run.

[ligaz]

@Granluke Can you recompile with this patch (#1925) and check is it working on your side?

[lida2003]

I met the same issue: https://forums.developer.nvidia.com/t/segnet-posenet-segmentation-fault-core-dumped-for-jetpack6-2/321496