Skip to content

Commit 895a2f4

Browse files
PlaidCatPlaidCat
committed
tcp: fix mptcp DSS corruption due to large pmtu xmit
jira LE-2522 Rebuild_History Non-Buildable kernel-5.14.0-503.29.1.el9_5 commit-author Paolo Abeni <[email protected]> commit 4dabcdf Empty-Commit: Cherry-Pick Conflicts during history rebuild. Will be included in final tarball splat. Ref for failed cherry-pick at: ciq/ciq_backports/kernel-5.14.0-503.29.1.el9_5/4dabcdf5.failed Syzkaller was able to trigger a DSS corruption: TCP: request_sock_subflow_v4: Possible SYN flooding on port [::]:20002. Sending cookies. ------------[ cut here ]------------ WARNING: CPU: 0 PID: 5227 at net/mptcp/protocol.c:695 __mptcp_move_skbs_from_subflow+0x20a9/0x21f0 net/mptcp/protocol.c:695 Modules linked in: CPU: 0 UID: 0 PID: 5227 Comm: syz-executor350 Not tainted 6.11.0-syzkaller-08829-gaf9c191ac2a0 #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 08/06/2024 RIP: 0010:__mptcp_move_skbs_from_subflow+0x20a9/0x21f0 net/mptcp/protocol.c:695 Code: 0f b6 dc 31 ff 89 de e8 b5 dd ea f5 89 d8 48 81 c4 50 01 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 cc cc cc cc e8 98 da ea f5 90 <0f> 0b 90 e9 47 ff ff ff e8 8a da ea f5 90 0f 0b 90 e9 99 e0 ff ff RSP: 0018:ffffc90000006db8 EFLAGS: 00010246 RAX: ffffffff8ba9df18 RBX: 00000000000055f0 RCX: ffff888030023c00 RDX: 0000000000000100 RSI: 00000000000081e5 RDI: 00000000000055f0 RBP: 1ffff110062bf1ae R08: ffffffff8ba9cf12 R09: 1ffff110062bf1b8 R10: dffffc0000000000 R11: ffffed10062bf1b9 R12: 0000000000000000 R13: dffffc0000000000 R14: 00000000700cec61 R15: 00000000000081e5 FS: 000055556679c380(0000) GS:ffff8880b8600000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000020287000 CR3: 0000000077892000 CR4: 00000000003506f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: <IRQ> move_skbs_to_msk net/mptcp/protocol.c:811 [inline] mptcp_data_ready+0x29c/0xa90 net/mptcp/protocol.c:854 subflow_data_ready+0x34a/0x920 net/mptcp/subflow.c:1490 tcp_data_queue+0x20fd/0x76c0 net/ipv4/tcp_input.c:5283 tcp_rcv_established+0xfba/0x2020 net/ipv4/tcp_input.c:6237 tcp_v4_do_rcv+0x96d/0xc70 net/ipv4/tcp_ipv4.c:1915 tcp_v4_rcv+0x2dc0/0x37f0 net/ipv4/tcp_ipv4.c:2350 ip_protocol_deliver_rcu+0x22e/0x440 net/ipv4/ip_input.c:205 ip_local_deliver_finish+0x341/0x5f0 net/ipv4/ip_input.c:233 NF_HOOK+0x3a4/0x450 include/linux/netfilter.h:314 NF_HOOK+0x3a4/0x450 include/linux/netfilter.h:314 __netif_receive_skb_one_core net/core/dev.c:5662 [inline] __netif_receive_skb+0x2bf/0x650 net/core/dev.c:5775 process_backlog+0x662/0x15b0 net/core/dev.c:6107 __napi_poll+0xcb/0x490 net/core/dev.c:6771 napi_poll net/core/dev.c:6840 [inline] net_rx_action+0x89b/0x1240 net/core/dev.c:6962 handle_softirqs+0x2c5/0x980 kernel/softirq.c:554 do_softirq+0x11b/0x1e0 kernel/softirq.c:455 </IRQ> <TASK> __local_bh_enable_ip+0x1bb/0x200 kernel/softirq.c:382 local_bh_enable include/linux/bottom_half.h:33 [inline] rcu_read_unlock_bh include/linux/rcupdate.h:919 [inline] __dev_queue_xmit+0x1764/0x3e80 net/core/dev.c:4451 dev_queue_xmit include/linux/netdevice.h:3094 [inline] neigh_hh_output include/net/neighbour.h:526 [inline] neigh_output include/net/neighbour.h:540 [inline] ip_finish_output2+0xd41/0x1390 net/ipv4/ip_output.c:236 ip_local_out net/ipv4/ip_output.c:130 [inline] __ip_queue_xmit+0x118c/0x1b80 net/ipv4/ip_output.c:536 __tcp_transmit_skb+0x2544/0x3b30 net/ipv4/tcp_output.c:1466 tcp_transmit_skb net/ipv4/tcp_output.c:1484 [inline] tcp_mtu_probe net/ipv4/tcp_output.c:2547 [inline] tcp_write_xmit+0x641d/0x6bf0 net/ipv4/tcp_output.c:2752 __tcp_push_pending_frames+0x9b/0x360 net/ipv4/tcp_output.c:3015 tcp_push_pending_frames include/net/tcp.h:2107 [inline] tcp_data_snd_check net/ipv4/tcp_input.c:5714 [inline] tcp_rcv_established+0x1026/0x2020 net/ipv4/tcp_input.c:6239 tcp_v4_do_rcv+0x96d/0xc70 net/ipv4/tcp_ipv4.c:1915 sk_backlog_rcv include/net/sock.h:1113 [inline] __release_sock+0x214/0x350 net/core/sock.c:3072 release_sock+0x61/0x1f0 net/core/sock.c:3626 mptcp_push_release net/mptcp/protocol.c:1486 [inline] __mptcp_push_pending+0x6b5/0x9f0 net/mptcp/protocol.c:1625 mptcp_sendmsg+0x10bb/0x1b10 net/mptcp/protocol.c:1903 sock_sendmsg_nosec net/socket.c:730 [inline] __sock_sendmsg+0x1a6/0x270 net/socket.c:745 ____sys_sendmsg+0x52a/0x7e0 net/socket.c:2603 ___sys_sendmsg net/socket.c:2657 [inline] __sys_sendmsg+0x2aa/0x390 net/socket.c:2686 do_syscall_x64 arch/x86/entry/common.c:52 [inline] do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83 entry_SYSCALL_64_after_hwframe+0x77/0x7f RIP: 0033:0x7fb06e9317f9 Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007ffe2cfd4f98 EFLAGS: 00000246 ORIG_RAX: 000000000000002e RAX: ffffffffffffffda RBX: 00007fb06e97f468 RCX: 00007fb06e9317f9 RDX: 0000000000000000 RSI: 0000000020000080 RDI: 0000000000000005 RBP: 00007fb06e97f446 R08: 0000555500000000 R09: 0000555500000000 R10: 0000555500000000 R11: 0000000000000246 R12: 00007fb06e97f406 R13: 0000000000000001 R14: 00007ffe2cfd4fe0 R15: 0000000000000003 </TASK> Additionally syzkaller provided a nice reproducer. The repro enables pmtu on the loopback device, leading to tcp_mtu_probe() generating very large probe packets. tcp_can_coalesce_send_queue_head() currently does not check for mptcp-level invariants, and allowed the creation of cross-DSS probes, leading to the mentioned corruption. Address the issue teaching tcp_can_coalesce_send_queue_head() about mptcp using the tcp_skb_can_collapse(), also reducing the code duplication. Fixes: 8571248 ("tcp: coalesce/collapse must respect MPTCP extensions") Cc: [email protected] Reported-by: [email protected] Closes: multipath-tcp/mptcp_net-next#513 Signed-off-by: Paolo Abeni <[email protected]> Acked-by: Matthieu Baerts (NGI0) <[email protected]> Signed-off-by: Matthieu Baerts (NGI0) <[email protected]> Link: https://patch.msgid.link/[email protected] Signed-off-by: Jakub Kicinski <[email protected]> (cherry picked from commit 4dabcdf) Signed-off-by: Jonathan Maple <[email protected]> # Conflicts: # net/ipv4/tcp_output.c
1 parent d4761f4 commit 895a2f4

File tree

1 file changed

+142
-0
lines changed

1 file changed

+142
-0
lines changed

‎ciq/ciq_backports/kernel-5.14.0-503.29.1.el9_5/4dabcdf5.failed

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
tcp: fix mptcp DSS corruption due to large pmtu xmit
2+
3+
jira LE-2522
4+
Rebuild_History Non-Buildable kernel-5.14.0-503.29.1.el9_5
5+
commit-author Paolo Abeni <[email protected]>
6+
commit 4dabcdf581217e60690467a37c956a5b8dbc6bd9
7+
Empty-Commit: Cherry-Pick Conflicts during history rebuild.
8+
Will be included in final tarball splat. Ref for failed cherry-pick at:
9+
ciq/ciq_backports/kernel-5.14.0-503.29.1.el9_5/4dabcdf5.failed
10+
11+
Syzkaller was able to trigger a DSS corruption:
12+
13+
TCP: request_sock_subflow_v4: Possible SYN flooding on port [::]:20002. Sending cookies.
14+
------------[ cut here ]------------
15+
WARNING: CPU: 0 PID: 5227 at net/mptcp/protocol.c:695 __mptcp_move_skbs_from_subflow+0x20a9/0x21f0 net/mptcp/protocol.c:695
16+
Modules linked in:
17+
CPU: 0 UID: 0 PID: 5227 Comm: syz-executor350 Not tainted 6.11.0-syzkaller-08829-gaf9c191ac2a0 #0
18+
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 08/06/2024
19+
RIP: 0010:__mptcp_move_skbs_from_subflow+0x20a9/0x21f0 net/mptcp/protocol.c:695
20+
Code: 0f b6 dc 31 ff 89 de e8 b5 dd ea f5 89 d8 48 81 c4 50 01 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 cc cc cc cc e8 98 da ea f5 90 <0f> 0b 90 e9 47 ff ff ff e8 8a da ea f5 90 0f 0b 90 e9 99 e0 ff ff
21+
RSP: 0018:ffffc90000006db8 EFLAGS: 00010246
22+
RAX: ffffffff8ba9df18 RBX: 00000000000055f0 RCX: ffff888030023c00
23+
RDX: 0000000000000100 RSI: 00000000000081e5 RDI: 00000000000055f0
24+
RBP: 1ffff110062bf1ae R08: ffffffff8ba9cf12 R09: 1ffff110062bf1b8
25+
R10: dffffc0000000000 R11: ffffed10062bf1b9 R12: 0000000000000000
26+
R13: dffffc0000000000 R14: 00000000700cec61 R15: 00000000000081e5
27+
FS: 000055556679c380(0000) GS:ffff8880b8600000(0000) knlGS:0000000000000000
28+
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
29+
CR2: 0000000020287000 CR3: 0000000077892000 CR4: 00000000003506f0
30+
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
31+
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
32+
Call Trace:
33+
<IRQ>
34+
move_skbs_to_msk net/mptcp/protocol.c:811 [inline]
35+
mptcp_data_ready+0x29c/0xa90 net/mptcp/protocol.c:854
36+
subflow_data_ready+0x34a/0x920 net/mptcp/subflow.c:1490
37+
tcp_data_queue+0x20fd/0x76c0 net/ipv4/tcp_input.c:5283
38+
tcp_rcv_established+0xfba/0x2020 net/ipv4/tcp_input.c:6237
39+
tcp_v4_do_rcv+0x96d/0xc70 net/ipv4/tcp_ipv4.c:1915
40+
tcp_v4_rcv+0x2dc0/0x37f0 net/ipv4/tcp_ipv4.c:2350
41+
ip_protocol_deliver_rcu+0x22e/0x440 net/ipv4/ip_input.c:205
42+
ip_local_deliver_finish+0x341/0x5f0 net/ipv4/ip_input.c:233
43+
NF_HOOK+0x3a4/0x450 include/linux/netfilter.h:314
44+
NF_HOOK+0x3a4/0x450 include/linux/netfilter.h:314
45+
__netif_receive_skb_one_core net/core/dev.c:5662 [inline]
46+
__netif_receive_skb+0x2bf/0x650 net/core/dev.c:5775
47+
process_backlog+0x662/0x15b0 net/core/dev.c:6107
48+
__napi_poll+0xcb/0x490 net/core/dev.c:6771
49+
napi_poll net/core/dev.c:6840 [inline]
50+
net_rx_action+0x89b/0x1240 net/core/dev.c:6962
51+
handle_softirqs+0x2c5/0x980 kernel/softirq.c:554
52+
do_softirq+0x11b/0x1e0 kernel/softirq.c:455
53+
</IRQ>
54+
<TASK>
55+
__local_bh_enable_ip+0x1bb/0x200 kernel/softirq.c:382
56+
local_bh_enable include/linux/bottom_half.h:33 [inline]
57+
rcu_read_unlock_bh include/linux/rcupdate.h:919 [inline]
58+
__dev_queue_xmit+0x1764/0x3e80 net/core/dev.c:4451
59+
dev_queue_xmit include/linux/netdevice.h:3094 [inline]
60+
neigh_hh_output include/net/neighbour.h:526 [inline]
61+
neigh_output include/net/neighbour.h:540 [inline]
62+
ip_finish_output2+0xd41/0x1390 net/ipv4/ip_output.c:236
63+
ip_local_out net/ipv4/ip_output.c:130 [inline]
64+
__ip_queue_xmit+0x118c/0x1b80 net/ipv4/ip_output.c:536
65+
__tcp_transmit_skb+0x2544/0x3b30 net/ipv4/tcp_output.c:1466
66+
tcp_transmit_skb net/ipv4/tcp_output.c:1484 [inline]
67+
tcp_mtu_probe net/ipv4/tcp_output.c:2547 [inline]
68+
tcp_write_xmit+0x641d/0x6bf0 net/ipv4/tcp_output.c:2752
69+
__tcp_push_pending_frames+0x9b/0x360 net/ipv4/tcp_output.c:3015
70+
tcp_push_pending_frames include/net/tcp.h:2107 [inline]
71+
tcp_data_snd_check net/ipv4/tcp_input.c:5714 [inline]
72+
tcp_rcv_established+0x1026/0x2020 net/ipv4/tcp_input.c:6239
73+
tcp_v4_do_rcv+0x96d/0xc70 net/ipv4/tcp_ipv4.c:1915
74+
sk_backlog_rcv include/net/sock.h:1113 [inline]
75+
__release_sock+0x214/0x350 net/core/sock.c:3072
76+
release_sock+0x61/0x1f0 net/core/sock.c:3626
77+
mptcp_push_release net/mptcp/protocol.c:1486 [inline]
78+
__mptcp_push_pending+0x6b5/0x9f0 net/mptcp/protocol.c:1625
79+
mptcp_sendmsg+0x10bb/0x1b10 net/mptcp/protocol.c:1903
80+
sock_sendmsg_nosec net/socket.c:730 [inline]
81+
__sock_sendmsg+0x1a6/0x270 net/socket.c:745
82+
____sys_sendmsg+0x52a/0x7e0 net/socket.c:2603
83+
___sys_sendmsg net/socket.c:2657 [inline]
84+
__sys_sendmsg+0x2aa/0x390 net/socket.c:2686
85+
do_syscall_x64 arch/x86/entry/common.c:52 [inline]
86+
do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
87+
entry_SYSCALL_64_after_hwframe+0x77/0x7f
88+
RIP: 0033:0x7fb06e9317f9
89+
Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
90+
RSP: 002b:00007ffe2cfd4f98 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
91+
RAX: ffffffffffffffda RBX: 00007fb06e97f468 RCX: 00007fb06e9317f9
92+
RDX: 0000000000000000 RSI: 0000000020000080 RDI: 0000000000000005
93+
RBP: 00007fb06e97f446 R08: 0000555500000000 R09: 0000555500000000
94+
R10: 0000555500000000 R11: 0000000000000246 R12: 00007fb06e97f406
95+
R13: 0000000000000001 R14: 00007ffe2cfd4fe0 R15: 0000000000000003
96+
</TASK>
97+
98+
Additionally syzkaller provided a nice reproducer. The repro enables
99+
pmtu on the loopback device, leading to tcp_mtu_probe() generating
100+
very large probe packets.
101+
102+
tcp_can_coalesce_send_queue_head() currently does not check for
103+
mptcp-level invariants, and allowed the creation of cross-DSS probes,
104+
leading to the mentioned corruption.
105+
106+
Address the issue teaching tcp_can_coalesce_send_queue_head() about
107+
mptcp using the tcp_skb_can_collapse(), also reducing the code
108+
duplication.
109+
110+
Fixes: 85712484110d ("tcp: coalesce/collapse must respect MPTCP extensions")
111+
Cc: [email protected]
112+
Reported-by: [email protected]
113+
Closes: https://github.com/multipath-tcp/mptcp_net-next/issues/513
114+
Signed-off-by: Paolo Abeni <[email protected]>
115+
Acked-by: Matthieu Baerts (NGI0) <[email protected]>
116+
Signed-off-by: Matthieu Baerts (NGI0) <[email protected]>
117+
Link: https://patch.msgid.link/[email protected]
118+
Signed-off-by: Jakub Kicinski <[email protected]>
119+
(cherry picked from commit 4dabcdf581217e60690467a37c956a5b8dbc6bd9)
120+
Signed-off-by: Jonathan Maple <[email protected]>
121+
122+
# Conflicts:
123+
# net/ipv4/tcp_output.c
124+
diff --cc net/ipv4/tcp_output.c
125+
index 26207b4fe4ff,68804fd01daf..000000000000
126+
--- a/net/ipv4/tcp_output.c
127+
+++ b/net/ipv4/tcp_output.c
128+
@@@ -2310,9 -2342,7 +2310,13 @@@ static bool tcp_can_coalesce_send_queue
129+
if (len <= skb->len)
130+
break;
131+
132+
++<<<<<<< HEAD
133+
+ if (unlikely(TCP_SKB_CB(skb)->eor) ||
134+
+ tcp_has_tx_tstamp(skb) ||
135+
+ !skb_pure_zcopy_same(skb, next))
136+
++=======
137+
+ if (tcp_has_tx_tstamp(skb) || !tcp_skb_can_collapse(skb, next))
138+
++>>>>>>> 4dabcdf58121 (tcp: fix mptcp DSS corruption due to large pmtu xmit)
139+
return false;
140+
141+
len -= skb->len;
142+
* Unmerged path net/ipv4/tcp_output.c

0 commit comments

Comments
 (0)