FW16 Freeze then Hang (FTH)

[jcdutton]

Device Information

System Model or SKU

[ ] Framework Laptop 16 (AMD Ryzen™ 7040 Series)
No dGPU.

BIOS VERSION

3.0.5

Windows:
N/A

Linux:
Open a terminal and run the following command
sudo dmidecode --string bios-version
03.05

DIY Edition information

Memory: Manufacture and SKU
Kingston Fury Impact: Part Number: KF556S40-32
2x making 64GB total.
Storage: Manufacture and SKU
Model Number: WD_BLACK SN850X 1000GB
Firmware Version: 620361WD

Port/Peripheral information

1. USB-C card, nothing plugged in.
2. Empty
3. Empty
4. Empty
5. USB-C card, FW16 PSU plugged in.
6. USB-A card, nothing plugged in.

Standalone Operation

Are you running your mainboard as a standalone device. Is standalone mode enabled in the BIOS?

• No

Describe the bug

This has only happened to me once so far.
The symptoms are:
Power plugged in.
Playing a video.
Laptop hangs, screen freezes, laptop plays audio in a short loop.
Wait 20 seconds, no automatic reboot.
Wait 60 seconds, still no reboot.
Force reboot by pressing the power button for 10 seconds.
No logs are stored, so no crash log is available.
No pstore crash log output.
No useful S5_RESET_STATUS, because I had to manually long press the power button to reset it.
EC port80 output runs over the 4096 log limit I had so no useful output captured their either.
I think that while it was in the "hang" state, it was still outputting port 80 output (keeps repeating the pattern):
Log index, Port 80 Value, Port 80 Value in ASCII or a decoded value.
"00005324","e825f022",".%.."
"00005325","e825f028",".%.("
"00005326","e825f90e",".%.."
"00005327","e825f90d",".%.."
"00005328","e825f90e",".%.."
"00005329","e825f90d",".%.."
"0000532a","e825f022",".%.."
...
Those are all values it also outputs when the laptop is running fine.
Long press of power button to power off gives:
"000054af","00001001","(S3->S0)"

Steps To Reproduce

Steps to reproduce the behavior:

1. Start from a powered off laptop.
2. Power on laptop
3. Wait a random amount of time. Play videos, netflix, youtube etc.
4. System freezes and does not self-reboot.

Note: I generally have the power plugged in most of the time. For all the FTH I have seen, the power was plugged in at the time.

Expected behavior

It should not randomly freeze then hang. (FTH)

Screenshots

N/A

Operating System (please complete the following information):

• OS/Distribution: Linux/Ubuntu
• Version: 24.04
• Linux Kernel Version: uname -a 6.13.7 <- Mainline compiled kernel.

Additional context

Add any other context about the problem here.

[CDRXavier]

Related? (Sync Flood)
#41 (comment)

[jcdutton]

Just saw another of these.
No extra info to report.
The (Sync Flood) one results in a automatic reboot after 20 seconds, so this is different, because there is no automatic reboot when this one happens.
I had disabled module "kvm", so no VMs were running, so this is not Virtual Machine related.

[sydneymeyer]

@jcdutton I assume you're using some form of disk encryption.. are you using luks/dm-crypt or perhaps SED/OPAL?

[sinatosk]

This is an issue I too am experiencing ( Linux ) and I'm not using luks/dm-crypt or SED/OPAL

Now using Linux 6.15-rc4 but this issue has been going on back starting with 6.12.0/6.12.1 and upgraded BIOS to 3.05 on 21st November 2024.

The common thing I've noticed with FTH and FTR ( #41 ) issues is none of these issues happen until I start changing between AC ( Framework 180W power adapter ) and DC ( battery ). One other saying the same back on the Framework community forum last time I checked

As long as I stay in AC after system is booted ( including suspend/resume ), all is good. based on what @jcdutton and some others have said, I think it's something to do with how the BIOS/EC is managing power state.

There was one bug in Linux sometime in 6.13 that caused a freeze but that was fixed and there were warnings/errors about it

The FTH/FTR, no warnings and/or errors and this has been going on since BIOS 3.05

I've disabled the battery extender since BIOS 3.04 and charge limit has been set to 80% since day one.

[jcdutton]

I am running my own firmware on the EC that fixes all the charge / discharge cycling bugs with the FW EC firmware. I wrote the bug fixes myself.
I observed the problem about once a month. It has not been a month yet, but since using my own EC firmware, I have not observed either the FTH or FTR problem.
For those interested, my EC Firmware changes and install instructions are here:
https://github.com/jcdutton/EmbeddedController/wiki

[jcdutton]

I have recently observed this problem again with my FW16 AMD 7840HS.
I cannot prove it 100% yet, but it appears to happen more often when the PSU is connected to port 4,5.
I used to use port 4,5 for PSU when I reported this problem initially, For whatever reason, I have been using port 1,2 for PSU, and did not see the problem. I recently switched back to port 4,5 and am seeing the problem again.

Has anyone else see any correlation between PSU port and the issue "FTH" appearing ?

For me it has often be with me playing a youtube video and then system hangs, and plays the audio in a loop from the speakers.

Another new observation:
While in "FTH" state, using EC CCD.
I see many "SM-RMI Error 4". About one a second, repeating.
"apreset" does nothing.
"apshutdown" does shut down the laptop, like switching it off.

Although the Capslock light toggles when pressed, no input from the keyboard reaches the CPU.
I know this because I have rigged "sysrq-h" to output a port80 code, and I don't see that code appear on the EC CCD.

So, the new ask is, does changing the port the PSU is plugged into change how often people see FTH ?

[sinatosk]

All the FTH/FTR issues I've had was using port 1 only or when using battery. I'll try 4 and 5 and get back to you on this

edit1: and is this with your EC firmware changes applied too?

edit2: I often find the FTR/FTH ( FTH more frequently ) happens when watching a video ( youtube mostly ) and CPU is fully loaded ( compiling software for example ), single monitor and it happens more frequently when multi monitor is active

[jcdutton]

I had another FTH today.
While in "FTH" state, using EC CCD.
I see many "SM-RMI Error 4". About one a second, repeating.

So, it seems that the CPU is not responding to anything the EC sends it over the SM-RMI bus when this problem shows itself.

[jcdutton]

I got a FTH today, while on battery. No PSU plugged in at all.
The same "SM-RMI Error 4" message on the EC console.

[sinatosk]

Maybe speaking too soon.

Using port 1, 4 and 5 so far and still getting FTH's ( not had any FTR's ) and doesn't feel much different

port 2 on the other hand ( so far ) no FTR or FTH. Compiling or running a game, watching youtube on secondary display, playing music, framework panel at 165Hz with VRR/Adaptive v-sync all at the same time

Sounding like a word vommit parrot here but I think I'm seeing a pattern ( triggering FTH/FTR ). I've noticed that if I have dual monitor ( second monitor is 3840x2160@60 ), youtube video playing in Firefox and doing something from moderate to heavy CPU usage ( I was running dota 2 spectating a game but compiling stuff works too ), system FTH's in less than 60 minutes most of the time but can be more than 60 minutes. Record is 7 minutes. This is giving higher chance to trigger FTR/FTH

Doing moderate to heavy CPU usage on it's own or youtube video on it's own and no dual monitor enabled, the system still FTH's but it usually days to weeks before it FTH's and the FTR issue is less common, like it barely happens

I'm using

Framework 16 AMD Ryzen 7 7840HS with Radeon 780M ( no dGPU )
RAM/Memory 32GB ( 2x16GB ) Framework
NVME 2280: Western Digital SN850X 2TB - Firmware 620361WD
NVME 2230: Western Digital SN770M 2TB - Firmware 731120WD
BIOS 3.05
Gentoo Linux 2.17 ( Linux 6.15.0 mainline realtime, compiled by clang 20.1.6 with march and mtune set to znver4 )
KDE Plasma 6.3.5 Wayland
Firefox 139.0.1 Wayland

• Port 1 - Framework audio module
• Port 2 - USB-C module - Framework 180W power adapter
• Port 3 - USB-C module - nothing plugged in
• Port 4 - USB-C module - USB-C to DisplayPort UGREEN 8K adapter with Dell P2145Q monitor
• Port 5 - USB-C module - nothing plugged in
• Port 6 - USB-C module - nothing plugged in