Kernel built-in PM is unreliable

[majek]

Pre-requisites

• A similar issue has not been reported before.
• mptcp.dev website does not cover my case.
• An up-to-date kernel is being used.
• This case is not fixed with the latest stable (or LTS) version listed on kernel.org

What did you do?

Pardon my lack of specificity, but I can't quite pinpoint this problem. I'm trying to run vanilla Linux as MPTCP client. I'm expecting "interactive" semantics - I want to get a reliable MPTCP connection over ever-changing network interfaces.

I'm running ubuntu 22, kernel 3.8. I have NetworkManager configured and I see ip mptcp endpoints ... subflow nicely configured when new network interface is created/enabled. And the entries correctly disappear on interface down. All other settings (sysctl's ) are default. I'm using in-kernel built-in path manager.

I manually create an MPTCP flow to a server, the server reports clear "Do not attempt to establish new subflows to this address and port" flag - allows client to reconnect. Initially I have two network interfaces - ethernet and Wi-Fi and I indeed see two underlying TCP subflows created.

When I yank out ethernet, the appropriate subflow disapears. When I plug it in it's often recreated. The same works for Wi-Fi.

It's all nice and shiny, but for an unknown reason this just stops working at some point. I don't know - maybe I'm yanking the ethetnet off too often? Or with too small delay? I'm sure it works (ping works) but at some point, the mptcp just seems to give up and fails to establish a new flow. I don't think this is about hitting "ip mptcp limits". Maybe there is some race-condition in the system, and it moves the MPTCP to a broken state.

I don't expect any resolution for this ticket, but I wanted to report it. Maybe it's a well known bug or feature of PM?

Any hints on how to debug built-in PM and understand why it just switches mptcp socket to a "broken" state (ie: gives up on establishing new subflows) would be helpful.

Marek

What happened?

Built-in PM seems to give up on an MPTCP socket at some point.

What did you expect to have?

I would expect MPTCP socket to stay working while I'm adding/removing network interfaces, as long as at least one network interface is always in a good state. I understand that when I break all connectivity MPTCP might, rightfully so, error.

System info: Client

Linux mrnew 6.8.0-49-generic #49-Ubuntu SMP PREEMPT_DYNAMIC Mon Nov  4 02:06:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
net.mptcp.add_addr_timeout = 120
net.mptcp.allow_join_initial_addr_port = 1
net.mptcp.checksum_enabled = 0
net.mptcp.close_timeout = 60
net.mptcp.enabled = 1
net.mptcp.pm_type = 1
net.mptcp.scheduler = default
net.mptcp.stale_loss_cnt = 4
192.168.1.149 id 13 subflow dev wlp0s20f3 
XXXXX id 14 subflow dev wlp0s20f3 
add_addr_accepted 0 subflows 2

System info: Server

.

Additional context

No response

[matttbe]

Hi @majek,

Thank you for this bug report.

Maybe it's a well known bug or feature of PM?

I think it is a known issue, maybe:

• an issue with NM, can you check the workaround please? https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/1583

• but also, it looks like all the fixes I did a few months ago around the in-kernel PM have not been backported to this Ubuntu kernel: Ubuntu 6.8 vs Linux 6.6.y Stable. (A bit sad to find out that...)

Could you eventually retry with an upstream kernel please? https://kernel.ubuntu.com/mainline/

[matttbe]

Any hints on how to debug built-in PM and understand why it just switches mptcp socket to a "broken" state (ie: gives up on establishing new subflows) would be helpful.

There are MIB counters (nstat) available in more recent versions. For this version, the "easiest" is to enable (some) pr_debug() from the net/mptcp/pm_netlink.c file.

[majek]

I upgraded to v6.12 and it seems way more reliable. thanks.

Now, this is probably another issue, but let me brain-dump here. The default PM works fine in the "Do not attempt to establish new subflows to this address and port" case. When I set this flag on the server (preventing reconnect) and ADD-ADDR a new address, the PM does unexpected stuff.

tcp   ESTAB      0      0       192.168.2.63:55377 x:12001
tcp   ESTAB      0      0       192.168.2.63:32872 x:12000
mptcp ESTAB      0      0       192.168.2.63:32872 x:12000

That is: I get two subflows, one original, and a new one to the ADD-ADDR signalled endpoint (port 12001) but both are from single iface/ip. Therefore not giving me any of the "interactive" benfits. I would expect at least one subflow to be from my wi-fi. BTW limts are high:

add_addr_accepted 8 subflows 8 

[majek]

The lack of sensible add-addr handling on the client is covered in #503

[matttbe]

• but also, it looks like all the fixes I did a few months ago around the in-kernel PM have not been backported to this Ubuntu kernel: Ubuntu 6.8 vs Linux 6.6.y Stable. (A bit sad to find out that...)

FYI, I talked to Ubuntu kernel devs today, and apparently, these fixes are coming, and they should be available in the next sync around February. Most of these fixes are already in their "master-next". We were just unlucky here because the last sync was done at the end of July, a few days before the fixes hit upstream stable...

That is: I get two subflows, one original, and a new one to the ADD-ADDR signalled endpoint (port 12001) but both are from single iface/ip. Therefore not giving me any of the "interactive" benfits. I would expect at least one subflow to be from my wi-fi.

That's probably the expected behaviour that is explained on mptcp.dev:

Upon the reception of an ADD_ADDR, and if the add_addr_accepted limits has not been reached yet, a new subflow will be created using the local address the routing configuration will pick, except if there are endpoints with the fullmesh flag. In this case, each endpoints with the fullmesh flag will be used to create a new subflow to the announced address.

In other words, there are workarounds to use different interfaces, but it might be cleaner to have a new dedicated flag as suggested in #503

[majek]

Just to get it right. On a linux MPTCP client, when ip mptcp limits are not being reached:

ip mptcp endpoint flag	address received via	subflow issued
subflow	SYN, allowed server ip/port reuse	YES
subflow	ADD-ADDR	NO
subflow, fullmesh	SYN, allowed server ip/port reuse	YES
subflow, fullmesh	ADD-ADDR	YES

Something like that? What about backup?

[matttbe]

Something like that?

Yes, correct. When an ADD-ADDR is received, first the in-kernel checks for fullmesh endpoints. If none are found, it asks to create a new subflow with IPADDRANY for the local address → so the source address will be selected after having looked at the routing table.

What about backup?

When used with subflow, it doesn't change anything on the number of subflows that are created. Technically, the subflow will be marked as "backup", and the packet scheduler will take that into consideration when selection on which path the next data will be sent.