Random machine freeze after attempting to use camera

[wallarug]

Hi there,

After updating my Raspberry Pi on 6th July to fix the DMA Heap Error (caused by dhcpd5) raspberrypi/rpicam-apps#809, I am now seeing a significant increase in system freezes and lock ups.

I've attached all the logs to this issue: raspberrypi/rpicam-apps#849

I don't have any specific logs that point to libcamera being the issue, however, it always seems to occur after the libcamera command (rpicam-still or libcamera-still) are launched.

It is random and inconsistent, other than it happening every 2-4 days.

What happens?

• The camera attempts to take an image.
• The systemd services all restart (including ones not related to the camera process)
• After about 30 - 40 seconds the whole system locks up and freezes, with the only way to recover is a hard-power cycle.
• On reboot, there is NO LOGGING indicating a problem at that time.
  • No low power warnings
  • No 'kernel panics' or logs
  • No system processes saying they are going to restart things
  • No logs after that 30 - 40 second window from any system processes

When the system is idle and not running the camera, the above crashes do not occur.

I'm at a loss as to why libcamera would be causing my whole system to soft-reset all the systemd processes and crash out my whole Raspberry Pi.

I've spent a month investigating this and I don't know what else to do but report an underlying unknown issue in the libcamera library and seek assistance from the experts.

Happy to provide any further explanation on this and try anything you think useful.

[naushir]

It's very unlikely that libcamera would cause a system lockup. This almost feels like a power supply issue since the system gets loaded significantly more when running a camera use case. Even though you don't see any no low-power warnings, it could still be a cause. Are you running with a standard Raspberry Pi power supply?

[wallarug]

It's very unlikely that libcamera would cause a system lockup. This almost feels like a power supply issue since the system gets loaded significantly more when running a camera use case. Even though you don't see any no low-power warnings, it could still be a cause. Are you running with a standard Raspberry Pi power supply?

Hey - this is happening on two of our cameras. One running the 5A official RPi PSU (it was happening with 3A PSU on this one as well). Another one running a custom power solution.

The camera running the custom power solution was stable before we updated libcamera/rpicam-apps on July 6th. Literally after that update we have now seen constant system lock ups every 2 to 4 days consistently. Nothing else has changed except that we updated libcamera/rpicam-apps.

I was thinking PSU issue originally to, but to have it happening on two completely separate systems hints at another issue. Additionally, it doesn't "power cycle" the Raspberry Pi. It just restarts all the processes, then locks up. There's no 'restart' registered in the log like you would normally see with a power lock up; and there was no warnings or logs other than the 'soft system process restarts' 30 seconds prior (which is super strange...).

I appreciate the view that libcamera shouldn't lock up the whole system. But from experience with this for the past two months and the DMA Heap Memory issue caused by dhcpd5 - I am questioning everything since the update.

Do you have any suggestions for how best to catch a power related event to prove that it is the PSU failing?