Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

XDP synproxy acceleration with iptables SYNPROXY cause small packet transmission intermittently #7

Open
vincentmli opened this issue Nov 7, 2024 · 4 comments
Open

XDP synproxy acceleration with iptables SYNPROXY cause small packet transmission intermittently #7

vincentmli opened this issue Nov 7, 2024 · 4 comments
Assignees
@vincentmli

Comments

@vincentmli
Copy link
Owner

vincentmli commented Nov 7, 2024 •
edited
Loading

when test loading simple website https://www.bpfire.net from https://check-host.net/check-http?host=https://www.bpfire.net, there are 5-6 seconds delay for some country locations when XDP syncookie program is attached to the bpfire.net server interface.

from the capture on client machine, it looks bpfire.net server send small size packet Len=501 according to client window size 501, not using client's caculated window size 64128 according to window size scaling factor

Screenshot 2024-11-07 at 3 41 35 PM
@vincentmli
Copy link
Owner Author

another capture showing the small packet size sent from server

Screenshot 2024-11-08 at 11 02 30 AM

vincentmli added a commit that referenced this issue Nov 10, 2024
@vincentmli
xdp-synproxy: code optimization to reduce latency
0659e96 
the XDP synproxy program from kernel selftest
seems aiming to test the correctness of BPF infrastructure.
not necessarily aiming for production code efficiency,
and production performance. Make a few code optimizations
to reduce real production latency.

see #7

Signed-off-by: Vincent Li <[email protected]>
@vincentmli
Copy link
Owner Author

vincentmli commented Nov 10, 2024 •
edited
Loading

the small packet transmission happens intermittently with https://check-host.net/check-http?host=https://www.bpfire.net for some locations up to 4 - 6 seconds delay when XDP generated SYNACK with tcp option mss, wscale, SACK, timestamp enabled. only enable mss option for the XDP generated SYNACK, the small packet transmission issue does not occur, use 72cd8df as workaround for now

vincentmli added a commit to vincentmli/BPFire that referenced this issue Nov 10, 2024
@vincentmli
xdp-synproxy: correct tcp header length
5b4593a 
the tcp header length is wrong and potentially
result in small packet transmission and result
in slow website page loading when XDP synproxy
program is attached.

also made a few code optimizations. see [0]

[0]: vincentmli/xdp-tools#7

Signed-off-by: Vincent Li <[email protected]>
@vincentmli vincentmli changed the title xdp_synproxy.bpf.c cause website loading delay xdp_synproxy.bpf.c cause small packet transmission and website loading delay Nov 10, 2024
@vincentmli vincentmli changed the title xdp_synproxy.bpf.c cause small packet transmission and website loading delay xdp_synproxy.bpf.c cause small packet transmission and delay website loading Nov 10, 2024
@vincentmli vincentmli self-assigned this Nov 10, 2024
vincentmli added a commit that referenced this issue Nov 10, 2024
@vincentmli
xdp-synproxy: code optimization to reduce latency
5aff083 
the XDP synproxy program from kernel selftest
seems aiming to test the correctness of BPF infrastructure.
not necessarily aiming for production code efficiency,
and production performance. Make a few code optimizations
to reduce real production traffic latency.

see #7

Co-developed-by: "DNSPROXY.ORG LLC <[email protected]>"
Signed-off-by: Vincent Li <[email protected]>
vincentmli added a commit to vincentmli/BPFire that referenced this issue Nov 10, 2024
@vincentmli
xdp-synproxy: bpf_loop with number tcp options
59d7379 
bpf_loop is hard coded with 6, this may result
in tcp options not parsing correctly ? and result
in small packet transmission to slow down website
page loading when XDP synproxy program is attached.

also the for loops introduce extra execution time.
potentially affect real production traffic. see [0]

[0]: vincentmli/xdp-tools#7

Signed-off-by: Vincent Li <[email protected]>
vincentmli added a commit that referenced this issue Nov 10, 2024
@vincentmli
xdp-synproxy: bpf_loop with number of tcp options
c2823e5 
the XDP synproxy program from kernel selftest
seems aiming to test the correctness of BPF infrastructure.
not necessarily aiming for production code efficiency,
and production performance. bpf_loop is hard coded with loop
count 6, this may result in tcp options not parsing correctly
in real production traffic and potentially cause traffic latency
with small packet being sent over the wire.

see #7

Reported-by: "DNSPROXY.ORG LLC <[email protected]>"
Signed-off-by: Vincent Li <[email protected]>
vincentmli added a commit to vincentmli/BPFire that referenced this issue Nov 11, 2024
@vincentmli
xdp-synproxy: bpf_loop with number tcp options
f3e04e9 
bpf_loop is hard coded with 6, this may result
in tcp options not parsing correctly for real
world production traffic and result in small
packet transmission to slow down website
page loading.

also the for loops introduce extra execution time.
potentially affect real production traffic. see [0]

[0]: vincentmli/xdp-tools#7

Reported-by: DNSPROXY.ORG LLC <[email protected]>
Signed-off-by: Vincent Li <[email protected]>
vincentmli added a commit that referenced this issue Nov 11, 2024
@vincentmli
xdp-synproxy: allow enable disable wscale option
72cd8df 
this is to workaround XDP synproxy intermittent small packet
transmission issue when window scale, time stamp is enabled

see #7

./xdp_synproxy --iface lo --mss4 1460 --mss6 1440  --wscale 0 --ttl 64 --ports 443

result XDP generated SYNACK TCP Options: (4 bytes), Maximum segment size

./xdp_synproxy --iface lo --mss4 1460 --mss6 1440  --wscale 7 --ttl 64 --ports 443

result XDP generated SYNACK TCP Options:
(16 bytes), Maximum segment size, SACK permitted, Timestamps, NOP, Wscale

Signed-off-by: Vincent Li <[email protected]>
vincentmli added a commit to vincentmli/BPFire that referenced this issue Nov 12, 2024
@vincentmli
xdp-synproxy: enable or disable window scaling
5de3f44 
XDP generated SYNACK tcp options with window
scaling and timestamp could intermittently cause
small packet transmission on DDoS protected server.
allow user to disable window scaling when such
problem occurs. see [0]

[0]: vincentmli/xdp-tools#7

Reported-by: DNSPROXY.ORG LLC <[email protected]>
Signed-off-by: Vincent Li <[email protected]>
@vincentmli
Copy link
Owner Author

vincentmli commented Nov 12, 2024 •
edited
Loading

further testing also shows that if do not attach the XDP program, but only setup the iptables rules below, the small packet size transmission problem also occurs. remove the --sack-perm --timestamp --wscale 7 --mss 1460 from the iptables rules, no small packet size transmission problem.

#!/bin/bash

INTERFACE=eth0

sysctl -w net.ipv4.tcp_syncookies=2
sysctl -w net.ipv4.tcp_timestamps=1
sysctl -w net.netfilter.nf_conntrack_tcp_loose=0
iptables -t raw -I PREROUTING  -i $INTERFACE -p tcp -m tcp --syn --dport 443 -j CT --notrack
iptables -t filter -A INPUT -i $INTERFACE -p tcp -m tcp -m state --state INVALID,UNTRACKED --dport 443 -j SYNPROXY --sack-perm --timestamp --wscale 7 --mss 1460

but if attach the XDP program with iptables rules without --sack-perm --timestamp --wscale 7 --mss 1460, the website becomes unavailable.

@vincentmli vincentmli changed the title xdp_synproxy.bpf.c cause small packet transmission and delay website loading XDP synproxy acceleration with iptables SYNPROXY cause small packet transmission intermittently Nov 12, 2024
@vincentmli
Copy link
Owner Author

sysctl -w net.ipv4.tcp_syncookies=1 alone also solves the issue if only use iptables SYNPROXY module without XDP acceleration

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vincentmli vincentmli

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@vincentmli