Cannot connect to ZMK keyboard over Bluetooth on Pop_OS! 22.04 + Intel AX200 controller

[sethforprivacy]

Hi all,

For some reason (I believe after updating from Pop_OS! 20.10 to 22.04), I am no longer able to connect to either of my ZMK firmware keyboards over Bluetooth. I've collected all the logs I can think of below:

Bluetoothctl output:
Failed to pair: org.bluez.Error.AuthenticationFailed

dmesg output:
[ 9738.118236] Bluetooth: hci0: unexpected SMP command 0x0b from e1:ed:df:d8:a0:25
[ 9738.232020] Bluetooth: hci0: Opcode 0x 401 failed: -16

systemctl journal output:
src/device.c:load_gatt_db() No cache for E1:ED:DF:D8:A0:25

USB logging output:
[00:03:08.153,594] <dbg> zmk: zmk_endpoints_send_report: usage page 0x07
[00:03:19.176,818] <dbg> zmk: connected: Connected thread: 0x20004d90
[00:03:19.176,910] <dbg> zmk: connected: Connected 44:AF:28:56:A6:C0 (public)
[00:03:19.178,405] <dbg> zmk: update_advertising: advertising from 0 to 2
[00:03:19.213,256] <inf> bas: BAS Notifications enabled
[00:03:19.300,628] <err> zmk: Security failed: 44:AF:28:56:A6:C0 (public) level 1 err 4
[00:03:19.390,991] <inf> bas: BAS Notifications disabled
[00:03:19.393,676] <dbg> zmk: disconnected: Disconnected from 44:AF:28:56:A6:C0 (public) (reason 0x05)
[00:03:19.393,707] <dbg> zmk: update_advertising: advertising from 2 to 2

The Bluetooth controller is the built-in controller in the Intel AX200 chipset. I've tested all things I can think of, including manually installing the latest linux-firmware, reinstalling Pop_OS! from scratch, etc.

The connection does work properly on Windows and on another Linux laptop, plus my Android phone.

https://paste.sethforprivacy.com/?d908b857102a6e0d#6Csv3EbvFSz7HeSGTNCzdJupZkze4ZxAmoqVs3fVa5Wr

[sethforprivacy]

A friend of mine more familiar with ZMK suggested clearing all Bluetooth profiles on the keyboard (not just the one I was trying to use) and after doing so the issue is resolved.

If I can provide any additional logs to help track down this bug, please let me know -- I will leave this issue open for a bit in case that would be helpful!

[Huckies]

Maybe the system upgrade wiped all the bonding informations, so the device can not be recognized anymore, and a sweep of Bluetooth profiles on keyboard may remove all the redundant bonds so they can be re-paired and re-bonded again.

[sethforprivacy]

Maybe the system upgrade wiped all the bonding informations, so the device can not be recognized anymore, and a sweep of Bluetooth profiles on keyboard may remove all the redundant bonds so they can be re-paired and re-bonded again.

The problem here is that the issue existed on two different ZMK-based keyboards, one of which was never connected before the upgrade to 22.04 as it didn't exist then.

I also cleared the first 4 profiles (the only ones I have ever tried to use on this keyboard) and that didn't resolve the issue, it was only resolved after clearing all profiles. Seems to be a bug of some sort that requires clearing all profiles from storage before you can properly connect in some scenarios.

[19216801-Franz]

I have the same recurring issue, running ZMK on two nice!nanos for my new split keyboard. Everytime I clear the profiles / reflash the firmware the issue goes away, but returns after one successful connected session using the keyboard over Bluetooth. Any Updates on this by anybody?

[Runeword]

I have the same issue on nixos 23.11 Failed to pair: org.bluez.Error.AuthenticationFailed

I have done all the following : Removing the device, clearing all bluetooth profiles and flashing a settings reset firmware.
After several attempts, I managed to pair the device in sudo mode.

But later I had to connect the device via usb to charge the battery, and after that the same error occur again when I try to pair the device.

[Runeword]

Finally, pairing with bluetuith instead of bluetoothctl resolved the issue in my case.
https://github.com/darkhz/bluetuith

[neyb]

Same issue on a glove80. Definitely a zmk strange behaviour I cannot understand (clear all only worked for me)

[AlexandreBossard]

Same issue, when upgraded to kernel 6.6.2. I tried clearing all profiles and reflashing, all I get is a connect and an immediate disconnect:

kernel: Bluetooth: hci0: unexpected SMP command 0x0b from eb:28:2c:3b:2e:ce
bluedevil-wizard[8227]: kf.bluezqt: PendingCall Error: "Authentication Canceled"

Edit: so the kernel is fine, the issue is related to the arch package linux-firmware-20231110.74158e7a-1. I have no idea who is to blame here.

[neyb]

happens to me again... this time clearing all profiles did not help... (kernel 6.6.3)

[Huckies]

From the Bluetooth protocol point of view:
Authentication failure is most likely to happen with a set of missing stored keys on central side, or to put it more plainly the host side (e.g. smartphones, PCs...). If the key is also/only missing on the peripheral side, central will initiate another pairing procedure to get a new set of keys so all the problems will gone.

From the keyboard view:
There must be a upper limit for the count of stored keys, when the limit is breached, keys can get overlapped, discarded... and a similar restriction can consist in each slot, so a clearing of all profiles is always promising.

[MangoIV]

I first thought this was a kernel issue (works fine on linux 6.1, which I pinned for the time being) but I'm now also using a low energy pointing device which works perfectly fine, I'm starting to think that it's at least partially the fault of ZMK that this does not work

[MangoIV]

• chip: nrf52840
• kernel: 6.6.2 zen, has not been working since 6.4

[MangoIV]

this is pretty serious, if ZMK doesn't support all linux kernels >6.4

[petejohanson]

this is pretty serious, if ZMK doesn't support all linux kernels >6.4

Can you get a full btmon dump during a failed connection, so we can see what's afoot?

[MangoIV]

I will try to do so @petejohanson do you know someone who is running a zmk keyboard on a newer kernel where it's working fine? I have had this issue on all my nrf52840 boards for a while now... I have also reported an issue in the nixos repository, some other folks also reported problems with BLE on newer kernel and wanted to take it to the kernel bug report, the issue is that now I'm using another device that is BLE and works just fine so I started thinking it's not purely the kernels fault.

[petejohanson]

I will try to do so @petejohanson do you know someone who is running a zmk keyboard on a newer kernel where it's working fine? I have had this issue on all my nrf52840 boards for a while now... I have also reported an issue in the nixos repository, some other folks also reported problems with BLE on newer kernel and wanted to take it to the kernel bug report, the issue is that now I'm using another device that is BLE and works just fine so I started thinking it's not purely the kernels fault.

I will check my recently updated Fedora laptop to see the specific kernel version, but otherwise I've not been tracking anything at that granularity at this point. I will need to do so if we see a trend.

Can you also try my Zephyr 3.5 PR to see if anything improves?

[MangoIV]

I have not updated zmk for a while, I will try this first. If issues persist, I will put logs here anyway.

This is first running connnect, then after this fails pair.

le-chiff-ble-connect-6-6-zen.log

[MangoIV]

issues persist on firmware built with the latest 3.5 container

[petejohanson]

issues persist on firmware built with the latest 3.5 container

Just to be clear, is it also built using the 3.5 Zephyr branch of ZMK?

[MangoIV]

https://hub.docker.com/layers/zmkfirmware/zmk-build-arm/3.5-branch-20231114235846-3.5.0-0.16.3-6c450465d1dd-6870957618/images/sha256-33cffa362f3613197c0c37788415271f534b80f168bb56650ba1419d2f03d46d?context=explore

I used this one...

[petejohanson]

https://hub.docker.com/layers/zmkfirmware/zmk-build-arm/3.5-branch-20231114235846-3.5.0-0.16.3-6c450465d1dd-6870957618/images/sha256-33cffa362f3613197c0c37788415271f534b80f168bb56650ba1419d2f03d46d?context=explore

I used this one...

You'd need to actually use the ZMK branch from #1995 and west update. Just using the new container won't use a new Zephyr version under the hood.

[MangoIV]

my github workflow runs west update; does the container not have the 3.5 zephyr? why was it named that?

[petejohanson]

my github workflow runs west update; does the container not have the 3.5 zephyr? why was it named that?

It includes all the SDK and python dependencies needed to build something targeting Zephyr 3.5.

[MangoIV]

NixOS/nixpkgs#242313 (comment)

[petejohanson]

I have not updated zmk for a while, I will try this first. If issues persist, I will put logs here anyway.

This is first running connnect, then after this fails pair.

le-chiff-ble-connect-6-6-zen.log

If this is logs from btmon then it's corrupted, perhaps.... Can we get btmon and dmesg output during the failure to connect?

[MangoIV]

btmon --read works fine for me, apparently it uses some binary format when doing --write

[petejohanson]

Another user reporting issues had a dmesg output indicating it was unhappy with the SMP security request packet... I do wonder if maybe using #2046 and enabling CONFIG_BT_GATT_AUTO_SEC_REQ=y will help.

[petejohanson]

btmon --read works fine for me, apparently it uses some binary format when doing --write

Ah, didn't know it had some binary format support. Looking now.

[petejohanson]

I also notice that the disconnects in both cases occur after the upgrade to 2M PHY... Perhaps related?

> HCI Event: LE Meta Event (0x3e) plen 6                                                                                                  #45 [hci0] 7.254650
      LE PHY Update Complete (0x0c)
        Status: Success (0x00)
        Handle: 16 Address: D7:4E:99:F1:BE:98 (Static)
        TX PHY: LE 2M (0x02)
        RX PHY: LE 2M (0x02)

[slabua]

Still no luck. And it is extremely problematic when the keyboard disconnects while typing as the last key stays stuck repeating itself for n times until it reconnects again.

[alistairmagee]

To those who have solved it with CONFIG_ZMK_BLE_EXPERIMENTAL_CONN=y, do you also have CONFIG_ZMK_BLE_EXPERIMENTAL_SEC=y?

I only used CONFIG_ZMK_BLE_EXPERIMENTAL_CONN=y as I read through the whole thread before attempting a fix and saw that recommended as a catch all. As it fixed the issue for me I didn't try anything else.

What does CONFIG_ZMK_BLE_EXPERIMENTAL_SEC=y do? Would it stop my desktop running KDE plasma from asking me to confirm the connection when I first pair or re-pair my keyboard?

[caksoylar]

What does CONFIG_ZMK_BLE_EXPERIMENTAL_SEC=y do? Would it stop my desktop running KDE plasma from asking me to confirm the connection when I first pair or re-pair my keyboard?

You can check it here: https://zmk.dev/docs/config/bluetooth#kconfig

It enables passkey entry requirement when pairing, where it asks you to enter a 6 digit pin and press enter. That sounds like what you are describing.

[alistairmagee]

This just broke again today. I don't know what changed between now and last night, I'm not aware of any software updates that happened on my laptop. I tried adding the CONFIG_ZMK_BLE_EXPERIMENTAL_SEC=y flag but as only one half connects to my computer, I can't input the security code, so I removed it again. Is this a problem with ZMK, the linux kernel, or Zephyr?

[petejohanson]

This just broke again today. I don't know what changed between now and last night, I'm not aware of any software updates that happened on my laptop. I tried adding the CONFIG_ZMK_BLE_EXPERIMENTAL_SEC=y flag but as only one half connects to my computer, I can't input the security code, so I removed it again. Is this a problem with ZMK, the linux kernel, or Zephyr?

You can use both halves of a split to enter the security code.

[alistairmagee]

This just broke again today. I don't know what changed between now and last night, I'm not aware of any software updates that happened on my laptop. I tried adding the CONFIG_ZMK_BLE_EXPERIMENTAL_SEC=y flag but as only one half connects to my computer, I can't input the security code, so I removed it again. Is this a problem with ZMK, the linux kernel, or Zephyr?

You can use both halves of a split to enter the security code.

When this bug occurs, the second half doesn't work. Even while primary half is connected via USB.

[petejohanson]

What is the keyboard hardware? Are you possibly fully draining the battery on one or both sides?

[alistairmagee]

Sofle split that I have made a permanent fixed base for (unsplit). They share power over a TRRS cable (permanently plugged in, no chance of damage from insertion and removal, keyboards can't move relative to each other and no room for cables to move) so when one side is powered both are powered.

…

On Tue, 13 Feb 2024 at 01:29, Pete Johanson ***@***.***> wrote: What is the keyboard hardware? Are you possibly fully draining the battery on one or both sides? — Reply to this email directly, view it on GitHub <#1487 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPNS32ZK7CRWDJUO3OAUSLYTIYOZAVCNFSM6AAAAAARBPH42KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZYHA4TQOJSGM> . You are receiving this because you commented.Message ID: ***@***.***>

[slabua]

As a note, perhaps the issue title should be changed. I guess a lot of people are having the very same issue with indeed different distros and/or configurations.

[Ay1tsMe]

Hoping there is some sort of fix for this. My keyboard which has two nice nano v2s is able to be paired with my android phone and another Linux laptop that has a AX201 Bluetooth controller. But it isn't even discoverable on my PC that has an AX200 controller. I went and bought another Bluetooth controller that was also an AX200 because I thought my old one was faulty but it still doesn't discover it so there must be some sort of Bluetooth issue with zmk connecting to these controllers. Hoping there is some sort of fix soon.

[lesshonor]

[My keyboard] isn't even discoverable on my PC

Your problem does not match the rest of the messages in this issue, then. All of the posts here involve pairing problems, not discovery problems.

[alistairmagee]

Some more points for troubleshooting.

I just got home from a long trip where I had to buy a cheap Bluetooth keyboard for my tablet as I couldn't get this fixed while on the road. I thought maybe that there was an actual error with my nice!nanos due to the weirdness I was having. I could connect the left hand, primary side to my computer with USB cable or Bluetooth. No communication between the halves would happen at this point. I actually had to change my keymap to add a way to get to my "raise" layer from left side to test things with clearing pairings and stuff.

I tried flashing each half with the other sides firmware (after flashing the reset firmware). This is what made me think it was a hardware issue as I could connect the "new" primary half via USB but not via Bluetooth. So maybe all my issues were with a n!n with broken bluetooth. I tested with my android phone to same results. Could connect to Left hand side when it was primary, couldn't connect to Right hand side. No communications between halves.

Back home on my desktop, connected via USB keyboard worked immediately. Both sides communicating as normal. Re-pairing the Bluetooth, it also appears to work fine until I tried changing Bluetooth profiles. Both profile 1 and 2, BT_SEL_0 and BT_SEL_1 were connected to the computer. Swapping between them changed nothing. Pressing BT_SEL_2 stopped input from the keyboard to the computer.

I tried pairing my tablet and it now paired correctly. I put in on slot 3, however it had the same behaviour. BT_SEL_0 would not stop input to the computer. I could swap computers by selecting slot 2 or 3, but slot 1 would just continue on the last computer connected.

This might actually be a hardware issue, but I'm writing everything at this point as I don't know what is going on.

Edit: Bluetooth profile issues are because that key switch was breaking. Ignore that.

TL;DR

1. Surface Go Tablet running Debian Linux, Bookworm, Gnome, Wayland, Kernel 6.7.2-surface-1 would connect to primary half of ZMK split but halves would not communicate with each other

2. Same issue on Android phone. No windows or Mac devices to test with at the time.

3. ZMK split does connect to Desktop PC running Debian Linux, Bookworm, KDE Plasma, (Edit 2: Actually first connected on wayland so this isn't a wayland bug) Xorg or Wayland, Kernel 6.1.0-13-amd64

4. However Bluetooth profile 1 appears to be working incorrectly.

5. After connecting to Desktop, keyboard connects to tablet again correctly.

Please tell me how I can help further.

My current keymap and conf files, in case I have done something obviously wrong and not noticed. I've enabled logging but I don't really know what I am doing with it.

[vvelarde]

Clearing all bt profiles worked for me. Thanks!

[iBot1986]

Hi,
I am not sure, this problem belongs to here;
I have some strange connection troubles with two different Windows machines and a nice!nano v 2.0 (Nordic nRF52840) custom keyboard. (no split keyboard).
Most of the time I must reconnect the keyboard on the computers (remove keyboard from Bluetooth devices and reconnect).
Sometimes it works like this, but sometimes I even must clear all Bluetooth profiles from the keyboard (BT_CLR_ALL) before I am able to reconnect.
The Keyboard has two multiple Bluetooth profiles available to select (BT_SEL 0-3).

I have tried with different config variations. Actually, I am going with CONFIG_ZMK_BLE_EXPERIMENTAL_FEATURES=y

Do you have any suggestions how to solve or debug this?

[slabua]

Still no luck with Fedora latest.

[sadekbaroudi]

So, thanks to @iBot1986
This issue is finally resolved for me. I haven't been able to use ZMK on two of my machines for years... I enabled the CONFIG_ZMK_BLE_EXPERIMENTAL_FEATURES option in my config, and it completely addressed the issue after entering the pairing security code. I realize that you posted it to note that it didn't help you in this case, but that was the issue for me. So I must thank you greatly for noting that in your comment.

This may be the issue for others as well, so I just wanted to also encourage others to try that.

[iBot1986]

@sadekbaroudi : Glad I could help!
I must admit that since I have written that comment, the problem haven't occur any more. And now I have even one additional Linux installation.
Maybe the problem caused by the more frequent flashes of the firmware, or also bad handling from my side...
Anyway, it seems CONFIG_ZMK_BLE_EXPERIMENTAL_FEATURES=y helped also for me.

[BigTunaKahuna]

Well have to admit it is much better now with CONFIG_ZMK_BLE_EXPERIMENTAL_FEATURES or CONFIG_ZMK_BLE_EXPERIMENTAL_SEC, not sure, I added both just in case, it's the one that makes the OS request a pin from the keyboard.

A pending issue in my case is when switching from Windows 11 to Fedora the connection is not auto-connecting on startup and I have to keep redoing the forget and add pin procedure, which usually does not work on the first try.

[lesshonor]

A pending issue in my case is when switching from Windows 11 to Fedora the connection is not auto-connecting on startup

If by this you mean you are dual-booting W11 and Fedora, you're going to need to take some extra steps to pair. See #2026 for previous discussion on the subject.

[dydyamotya]

Hello. I've encountered the same problem with connection on my ArchLinux fully updated machine. Using bluetoothctl I experienced the:
Authentification failed bluez error.
So, I've tried to add CONFIG_ZMK_BLE_EXPERIMENTAL_FEATURES=y and it pairs and works for now.

[kovasap]

CONFIG_ZMK_BLE_EXPERIMENTAL_FEATURES=y looks to have fixed my issue too. Although I didn't understand for a while that with that option I needed to manually type in the pairing code advertised by my bluetooth manager on my keyboard and press enter. Everything seems to work after doing that!

[vakochetkov]

If it helps someone. After adding CONFIG_ZMK_BLE_EXPERIMENTAL_FEATURES=y I was able to pair & connect my ZMK keyboard but failed to get any input from it.
sudo btmon showed one data message on first stroke and then silence, bluetoothd was throwing errors like

src/service.c:service_accept() input-hog profile accept failed for XX:XX:XX:XX:XX:XX

I almost gave up making it work but, found the workaround in bluez repo and my Ergounaut One keyboard works like a charm now!

P.S. I have a laptop with Intel 12th gen CPU and it seems many people using this CPU have troubles with ZMK. Another reason might be Arch Linux since used versions of bluez and kernel are little tested for now

[slabua]

I almost gave up making it work but, found the workaround

But my system already loads uhid successfully, still no dice :\

[Mauladen]

Thank you all for solving the problem, my keyboard successfully connected to Android and Windows 11, but not to ArchLinux, the solution with CONFIG_ZMK_BLE_EXPERIMENTAL_FEATURES=y helped me solve this problem, the main thing is to enter the pairing code