Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Syncd container exit on broadcom platform #21680

Open
philo-micas opened this issue Feb 8, 2025 · 6 comments
Open

Syncd container exit on broadcom platform #21680

philo-micas opened this issue Feb 8, 2025 · 6 comments
Labels
BRCM Triaged this issue has been triaged

Comments

@philo-micas
Copy link
Contributor

philo-micas commented Feb 8, 2025 •
edited
Loading

Broadcom platform, the sai seems not support 'SAI_API_PREFIX_COMPRESSION', API 53, the idx is out of range:

Failed log:

2024 Dec 13 16:46:15.081087 sonic DEBUG syncd#syncd: [none] SAI_API_UNSPECIFIED:sai_log_set:783 SAI API Log set  API type 53 level 2 idx -536870806
2024 Dec 13 16:46:15.084720 sonic INFO kernel: [  100.208616] syncd[17152]: segfault at 7fc05a1bfaa8 ip 00007fc0c84b11a5 sp 00007ffed4a76a30 error 6 in libsai.so.1.0[7fc0c82f8000+5e61000] likely on CPU 4 (core 0, socket 0)
2024 Dec 13 16:46:15.084741 sonic INFO kernel: [  100.208636] Code: 00 83 fe 05 0f 87 db 00 00 00 8d 9f 35 00 00 e0 83 ff 34 89 fd 0f 46 df 85 c0 74 29 83 fb 48 0f 8f 10 01 00 00 48 63 db 31 c0 <45> 89 64 9d 00 48 83 c4 08 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f
2024 Dec 13 16:46:15.219187 sonic INFO swss#supervisord 2024-12-13 16:46:15,218 INFO success: orchagent entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

version info

root@sonic:/home/admin# show version

SONiC Software Version: SONiC.master.766376-3fe020a9c
SONiC OS Version: 12
Distribution: Debian 12.9
Kernel: 6.1.0-22-2-amd64
Build commit: 3fe020a9c
Build date: Fri Feb  7 14:19:43 UTC 2025
Built by: azureuser@3a3b2860c000003

Platform: x86_64-micas_m2-w6930-64qc-r0
HwSKU: M2-W6930-64QC
ASIC: broadcom
ASIC Count: 1
Serial Number: G1SL123000364
Model Number: 01019AQ0
Hardware Revision: 100
Uptime: 06:23:04 up 5 min,  1 user,  load average: 1.72, 1.07, 0.49
Date: Sat 08 Feb 2025 06:23:04

Docker images:
REPOSITORY                    TAG                       IMAGE ID       SIZE
docker-syncd-brcm             latest                    5c6bf3c20e2b   763MB
docker-syncd-brcm             master.766376-3fe020a9c   5c6bf3c20e2b   763MB
docker-gbsyncd-broncos        latest                    465508714d8d   352MB
docker-gbsyncd-broncos        master.766376-3fe020a9c   465508714d8d   352MB
docker-gbsyncd-credo          latest                    170edc1430a1   326MB
docker-gbsyncd-credo          master.766376-3fe020a9c   170edc1430a1   326MB
docker-orchagent              latest                    66dcd9522a16   357MB
docker-orchagent              master.766376-3fe020a9c   66dcd9522a16   357MB
docker-nat                    latest                    5441b7601eb1   346MB
docker-nat                    master.766376-3fe020a9c   5441b7601eb1   346MB
docker-fpm-frr                latest                    7e1ed9895a65   378MB
docker-fpm-frr                master.766376-3fe020a9c   7e1ed9895a65   378MB
docker-dhcp-relay             latest                    8037aed12cc4   323MB
docker-macsec                 latest                    4a12779dbe7a   346MB
docker-snmp                   latest                    651a4a3be67e   358MB
docker-snmp                   master.766376-3fe020a9c   651a4a3be67e   358MB
docker-sonic-mgmt-framework   latest                    85e2ebfa0ea9   402MB
docker-sonic-mgmt-framework   master.766376-3fe020a9c   85e2ebfa0ea9   402MB
docker-platform-monitor       latest                    38cb7432fce4   434MB
docker-platform-monitor       master.766376-3fe020a9c   38cb7432fce4   434MB
docker-teamd                  latest                    a4c6303b2718   343MB
docker-teamd                  master.766376-3fe020a9c   a4c6303b2718   343MB
docker-sflow                  latest                    eccb3c745ff4   344MB
docker-sflow                  master.766376-3fe020a9c   eccb3c745ff4   344MB
docker-router-advertiser      latest                    ad464ef4d579   314MB
docker-router-advertiser      master.766376-3fe020a9c   ad464ef4d579   314MB
docker-lldp                   latest                    3e737872ae99   359MB
docker-lldp                   master.766376-3fe020a9c   3e737872ae99   359MB
docker-mux                    latest                    51139353e104   365MB
docker-mux                    master.766376-3fe020a9c   51139353e104   365MB
docker-sonic-gnmi             latest                    6c47de97b2be   404MB
docker-sonic-gnmi             master.766376-3fe020a9c   6c47de97b2be   404MB
docker-database               latest                    f0ef077e3c08   322MB
docker-database               master.766376-3fe020a9c   f0ef077e3c08   322MB
docker-sonic-bmp              latest                    6810413a3e83   315MB
docker-sonic-bmp              master.766376-3fe020a9c   6810413a3e83   315MB
docker-eventd                 latest                    9607374aff8b   314MB
docker-eventd                 master.766376-3fe020a9c   9607374aff8b   314MB

root@sonic:/home/admin# docker ps -a
CONTAINER ID   IMAGE                                COMMAND                  CREATED          STATUS                        PORTS     NAMES
7bbe405d6273   docker-sonic-gnmi:latest             "/usr/local/bin/supe…"   39 seconds ago   Up 38 seconds                           gnmi
741bf1e4e6cf   docker-sflow:latest                  "/usr/local/bin/supe…"   9 minutes ago    Exited (137) 8 minutes ago              sflow
e090521b0e5f   docker-platform-monitor:latest       "/usr/bin/docker_ini…"   9 minutes ago    Up 33 seconds                           pmon
b894252c28b4   docker-nat:latest                    "/usr/local/bin/supe…"   9 minutes ago    Exited (137) 6 minutes ago              nat
a39e9609bd43   docker-sonic-mgmt-framework:latest   "/usr/local/bin/supe…"   9 minutes ago    Up 35 seconds                           mgmt-framework
9051a8d44b27   docker-lldp:latest                   "/usr/bin/docker-lld…"   9 minutes ago    Up 36 seconds                           lldp
5544cc6fa20f   8037aed12cc4                         "/usr/bin/docker_ini…"   9 minutes ago    Exited (0) 26 seconds ago               dhcp_relay
c6b1ec4f6744   docker-fpm-frr:latest                "/usr/bin/docker_ini…"   10 minutes ago   Exited (137) 28 seconds ago             bgp
70bf80d91f12   docker-router-advertiser:latest      "/usr/bin/docker-ini…"   10 minutes ago   Exited (0) 39 seconds ago               radv
8ae7ac649659   docker-syncd-brcm:latest             "/usr/local/bin/supe…"   10 minutes ago   Exited (0) 24 seconds ago               syncd
9e16f33138cc   docker-teamd:latest                  "/usr/local/bin/supe…"   10 minutes ago   Exited (0) 41 seconds ago               teamd
0bfd1e0cc2f7   docker-orchagent:latest              "/usr/bin/docker-ini…"   10 minutes ago   Exited (0) 47 seconds ago               swss
0c9cbd848635   docker-eventd:latest                 "/usr/local/bin/supe…"   10 minutes ago   Up 5 minutes                            eventd
b8faeee18493   docker-database:latest               "/usr/local/bin/dock…"   10 minutes ago   Up 5 minutes
@philo-micas
Copy link
Contributor Author

syslog-1.txt

@ec-michael-shih
Copy link
Contributor

ec-michael-shih commented Feb 12, 2025 •
edited
Loading

Met same issue too.

2024 Dec  1 13:53:49.876167 sonic DEBUG syncd#syncd: [none] SAI_API_UNSPECIFIED:sai_log_set:783 SAI API Log set  API type 52 level 2 idx 52
2024 Dec  1 13:53:49.876167 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ICMP_ECHO
2024 Dec  1 13:53:49.876407 sonic DEBUG syncd#syncd: [none] SAI_API_UNSPECIFIED:sai_log_set:783 SAI API Log set  API type 53 level 2 idx -536870806
2024 Dec  1 13:53:49.876867 sonic INFO kernel: [ 1527.319890] syncd[21809]: segfault at 7ff969d05aa8 ip 00007ff9d7ff71a5 sp 00007ffd270932a0 error 6 in libsai.so.1.0[7ff9d7e3e000+5e61000] likely on CPU 2 (core 2, socket 0)
2024 Dec  1 13:53:49.876884 sonic INFO kernel: [ 1527.319907] Code: 00 83 fe 05 0f 87 db 00 00 00 8d 9f 35 00 00 e0 83 ff 34 89 fd 0f 46 df 85 c0 74 29 83 fb 48 0f 8f 10 01 00 00 48 63 db 31 c0 <45> 89 64 9d 00 48 83 c4 08 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f

syslog:
syslog-2.txt

root@sonic:~# show version

SONiC Software Version: SONiC.master.769885-d83b10455
SONiC OS Version: 12
Distribution: Debian 12.9
Kernel: 6.1.0-22-2-amd64
Build commit: d83b10455
Build date: Tue Feb 11 14:20:33 UTC 2025
Built by: azureuser@9e6be0a6c000000

Platform: x86_64-accton_as7726_32x-r0
HwSKU: Accton-AS7726-32X
ASIC: broadcom
ASIC Count: 1
Serial Number: 772632X1911006
Model Number: FP3ZZ7632074A
Hardware Revision: N/A
Uptime: 13:59:15 up 30 min,  2 users,  load average: 0.44, 0.61, 0.57
Date: Sun 01 Dec 2024 13:59:15

Docker images:
REPOSITORY                    TAG                       IMAGE ID       SIZE
docker-syncd-brcm             latest                    2f3690c3f1a8   763MB
docker-syncd-brcm             master.769885-d83b10455   2f3690c3f1a8   763MB
docker-gbsyncd-broncos        latest                    653c82d7f8a6   352MB
docker-gbsyncd-broncos        master.769885-d83b10455   653c82d7f8a6   352MB
docker-gbsyncd-credo          latest                    f4077413d552   326MB
docker-gbsyncd-credo          master.769885-d83b10455   f4077413d552   326MB
docker-orchagent              latest                    c985887d3ce4   357MB
docker-orchagent              master.769885-d83b10455   c985887d3ce4   357MB
docker-nat                    latest                    d98fddef158f   346MB
docker-nat                    master.769885-d83b10455   d98fddef158f   346MB
docker-dhcp-relay             latest                    7109cfb23b57   323MB
docker-fpm-frr                latest                    cb6a0416d89a   378MB
docker-fpm-frr                master.769885-d83b10455   cb6a0416d89a   378MB
docker-macsec                 latest                    574202e50815   346MB
docker-sonic-mgmt-framework   latest                    44074ab5ea92   402MB
docker-sonic-mgmt-framework   master.769885-d83b10455   44074ab5ea92   402MB
docker-snmp                   latest                    7e69f03d53a2   358MB
docker-snmp                   master.769885-d83b10455   7e69f03d53a2   358MB
docker-platform-monitor       latest                    bba58ab2dda4   434MB
docker-platform-monitor       master.769885-d83b10455   bba58ab2dda4   434MB
docker-teamd                  latest                    d602ffd229f1   343MB
docker-teamd                  master.769885-d83b10455   d602ffd229f1   343MB
docker-sflow                  latest                    176f0c3e3b22   344MB
docker-sflow                  master.769885-d83b10455   176f0c3e3b22   344MB
docker-router-advertiser      latest                    5ababe1f287f   314MB
docker-router-advertiser      master.769885-d83b10455   5ababe1f287f   314MB
docker-lldp                   latest                    18f092acfd18   359MB
docker-lldp                   master.769885-d83b10455   18f092acfd18   359MB
docker-mux                    latest                    929bed877c39   365MB
docker-mux                    master.769885-d83b10455   929bed877c39   365MB
docker-sonic-gnmi             latest                    ae06ac6ddc32   404MB
docker-sonic-gnmi             master.769885-d83b10455   ae06ac6ddc32   404MB
docker-database               latest                    9785e5dd0d08   322MB
docker-database               master.769885-d83b10455   9785e5dd0d08   322MB
docker-sonic-bmp              latest                    7affeac2466e   315MB
docker-sonic-bmp              master.769885-d83b10455   7affeac2466e   315MB
docker-eventd                 latest                    a13099fc9a64   314MB
docker-eventd                 master.769885-d83b10455   a13099fc9a64   314

@bingwang-ms
Copy link
Contributor

@adyeung Can you please take a look? Looks like a SAI implementation issue.

@bingwang-ms bingwang-ms added the Triaged this issue has been triaged label Feb 12, 2025
@adyeung
Copy link
Collaborator

adyeung commented Feb 13, 2025

Current latest SONiC master is on SAI header 1.15

sonic-net/sonic-sairedis#1507

SAI header 1.15.x does not have SAI_API_PREFIX_COMPRESSION 53 definition (despite it was merged 3 months ago), only the sai master has it

https://github.com/opencomputeproject/SAI/blob/v1.15.0/inc/sai.h

I am not sure how syncd is reaching type 53

@kcudnik
Copy link
Contributor

kcudnik commented Feb 13, 2025

this seems like brcm log_set issue, it's not supporting that index and probably getting out of array causing sigsegv

@adyeung
Copy link
Collaborator

adyeung commented Feb 14, 2025 •
edited
Loading

@kcudnik @prabhataravind the code PR below

sonic-net/sonic-sairedis#1507

moved SAI pairing to commit id 0540855, that contains SAI_API_PREFIX_COMPRESSION 53 in sai.h, see below

https://github.com/opencomputeproject/SAI/blob/054085547c2042b95ab0fa1a75a44da3288b26bf/inc/sai.h

However, official SAI 1.15.4 sai.h is not the same as 0540855

https://github.com/opencomputeproject/SAI/blob/v1.15.4/inc/sai.h

Is the intend to pair SONiC master with SAI 1.15.x or SAI Master? If it's the former, SAI commit id 0540855 pairing seems to be off.

sai_log_set() out-of-bound handling needs to be fixed separately, but please check what is the intended SAI header pairing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
BRCM Triaged this issue has been triaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@kcudnik @adyeung @bingwang-ms @ec-michael-shih @philo-micas