Skip to content

Latest commit

 

History

History
285 lines (214 loc) · 9.71 KB

benchmark.md

File metadata and controls

285 lines (214 loc) · 9.71 KB

How to benchmark / observe Sōzu

For debugging and optimization purposes.

Run Sōzu on your machine, with dummy backends

Create a config with an HTTP and an HTTPS listener

Most defaults of bin/config.toml are sensible, copy them into bin/my-config.toml. An important value to tweak is worker_count. Testing the performance of one worker only seems ideal:

worker_count = 1

The HTTP and HTTPS listeners should be identical. Chose a TLS version and a cipher list for the HTTPS listener, and stick to it.

[[listeners]]
protocol = "http"
address = "0.0.0.0:8080"

[[listeners]]
protocol = "https"
address = "0.0.0.0:8443"
tls_versions = ["TLS_V13"]

cipher_list = [
  # "TLS13_AES_128_GCM_SHA256",
  "TLS13_CHACHA20_POLY1305_SHA256",
  # ...
]

Create a cluster (equivalent to an app) that has two frontends: one for HTTP requests, one for HTTPS requests.

Make sure you created your own certificate and key for the localhost domain.

frontends = [
    { address = "0.0.0.0:8080", hostname = "localhost", path = "/api" },
    { address = "0.0.0.0:8443", hostname = "localhost", certificate = "/path/to/your/certificate.pem", key = "/path/to/your/key.pem", certificate_chain = "/path/to/your/certificate.pem" },
]

or use the certificate and key and certificate chain of config.toml, present in /lib/assets/. But then you have to use lolcatho.st as a local domain, and be sure to have this lines in your /etc/hosts:

127.0.0.1       lolcatho.st
::1             lolcatho.st

Spawn 4 backends

Create four simple HTTP servers that reply each 200 OK when requested with HTTP GET on the /api path (arbitrarily). Use your favorite programming langage, it should be simple. It's 8 lines of javascript. Make sure that each backend listens on localhost and on a different port, for exapmle: 1051, 1052, 1053 1054.

Add those backends to my-config.toml, under the frontends:

backends = [
    { address = "127.0.0.1:1051", backend_id = "backend_1" },
    { address = "127.0.0.1:1052", backend_id = "backend_2" },
    { address = "127.0.0.1:1053", backend_id = "backend_3" },
    { address = "127.0.0.1:1054", backend_id = "backend_4" },
]

Build and run Sōzu

Build Sōzu in release mode.

cargo build --release

Tip: rename the release binary and put it in the cargo path:

mv target/release/sozu $HOME/.cargo/bin/sozu-0.15.14

Now you can run Sōzu in release mode:

cd bin
sozu-0.15.14 --config my-config.toml start

Check that Sōzu works by curling the backends through Sōzu:

curl https://localhost:8443/api -v
curl http://localhost:8080/api  -v

Bombardier

Bombardier is an HTTP and HTTPS benchmarking tool written in go. It is available as a package in most linux distributions, and easy to use in the command line.

Test a lot of requests on one connection

bombardier --connections=1 --requests=10000 https://localhost:8443/api --latencies
bombardier --connections=1 --requests=10000 http://localhost:8080/api --latencies

Test concurrent connections

bombardier --connections=100 --requests=10000 https://localhost:8443/api --latencies
bombardier --connections=100 --requests=10000 http://localhost:8080/api --latencies

Increase the number of connections until Sõzu can't take it anymore.

TLS-perf

tls-perf is a TLS stress-test tool written in C. It has no dependencies and is easy to build.

git clone [email protected]:tempesta-tech/tls-perf.git
cd tls-perf
make
mv tls-perf $HOME/.local/bin

Assuming $HOME/.local/bin is in your path.

This command does:

  • ten thousand TLS handshakes
  • using only one connection
  • using TLS version 1.3
tls-perf \
  -n 10000 \
  -l 1 \
  --sni localhost \
  --tls 1.3 \
  127.0.0.1 8443

Find syscalls with strace

strace is a diagnostic tool that can monitor interactions between a process (Sōzu) and the linux kernel. It is available as a package in most linux distributions.

Once Sōzu runs, find the pid of the worker:

ps -aux | grep sozu
user   13368  0.7  0.1 188132 40344 pts/2    Sl+  11:35   0:00 /path/to/sozu --config my-config.toml start
user   14157  0.0  0.0  79908 29628 pts/2    S+   11:35   0:00 /path/to/sozu worker --id 0 --fd 5 --scm 7 --configuration-state-fd 3 --command-buffer-size 16384 --max-command-buffer-size 16384
user   14205  0.0  0.0   6556  2412 pts/3    S+   11:35   0:00 grep --color=auto sozu

That second line is the worker, its pid is 14157.

Use strace to track this pid. You may need root privileges.

strace --attach=14157

You should see regular epoll_wait syscalls (system calls), this is Sōzu's main loop.

epoll_wait(3, [], 1024, 1000)           = 0
epoll_wait(3, [], 1024, 1000)           = 0
epoll_wait(3, [], 1024, 1000)           = 0
epoll_wait(3, [], 1024, 318)            = 0

If you perform one of those curls, you will see all syscalls that Sōzu does during an HTTP or HTTPS request. Here are syscalls occuring during an HTTP GET on a simple backend. This is only the accepting of traffic on a listener socket.

# main loop waiting for events
epoll_wait(3, [], 1024, 1000)           = 0

# One readable event (EPOLLIN). A client connected to a listener socket.
epoll_wait(3, [{events=EPOLLIN, data={u32=3, u64=3}}], 1024, 1000) = 1

# The proxy accepts the connection, by telling a listener to accept
# incoming connections on the socket
accept4(9, {sa_family=AF_INET, sin_port=htons(46144), sin_addr=inet_addr("127.0.0.1")}, [128 => 16], SOCK_CLOEXEC|SOCK_NONBLOCK) = 11

# Done! No more connections to accept
accept4(9, 0x7ffe89e39b98, [128], SOCK_CLOEXEC|SOCK_NONBLOCK) = -1 EAGAIN (Resource temporarily unavailable)

# The HTTP proxy sets NODELAY on the socket
setsockopt(11, SOL_TCP, TCP_NODELAY, [1], 4) = 0

# Sōzu registers the socket in the accept queue
# The accept queue is popped in another loop, to create sessions

# When creating a session, the socket is registered in the event loop
epoll_ctl(6, EPOLL_CTL_ADD, 11, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=267, u64=267}}) = 0

# Get the address of the socket (here, the host is the same, but the port is different)
getpeername(11, {sa_family=AF_INET, sin_port=htons(46144), sin_addr=inet_addr("127.0.0.1")}, [128 => 16]) = 0

# Return to the main event loop
epoll_wait(3, [{events=EPOLLIN|EPOLLOUT, data={u32=267, u64=267}}], 1024, 1000) = 1

# We are ready for traffic to come through this socket

# Here it comes!
recvfrom(11, "GET /api HTTP/1.1\r\nHost: localho"..., 16400, 0, NULL, NULL) = 80

Find which parts of the code cause syscalls

For this kind of digging, Sōzu must run in debug mode, that is: not compiled with the --release flag. Do:

cargo run -- --config my-config.toml start

Find the pid as described above, and do:

strace --attach=14157 --stack-traces
# or
strace -p 14157 -k

Curl Sōzu once, for instance

curl http://localhost:8080/api  -v

Now you see what piece of code is responsible for each syscall. Here is, for instance, the code responsible for a connect syscall on a socket with file descriptor 12. We can see it is an HTTP session connecting to its backend:

connect(12, {sa_family=AF_INET, sin_port=htons(1052), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress)
 > /usr/lib/libc.so.6(__connect+0x14) [0x112454]
 > /path/to/sozu/target/debug/sozu(mio::sys::unix::tcp::connect+0x84) [0x17b2ec4]
 > /path/to/sozu/target/debug/sozu(mio::net::tcp::stream::TcpStream::connect+0x127) [0x17ad9e7]
 > /path/to/sozu/target/debug/sozu(sozu_lib::backends::Backend::try_connect+0x88) [0xb03898]
 > /path/to/sozu/target/debug/sozu(sozu_lib::backends::BackendMap::backend_from_cluster_id+0x465) [0xb04145]
 > /path/to/sozu/target/debug/sozu(sozu_lib::protocol::kawa_h1::Http<Front,L>::get_backend_for_sticky_session+0x5ab) [0xca8beb]
 > /path/to/sozu/target/debug/sozu(sozu_lib::protocol::kawa_h1::Http<Front,L>::backend_from_request+0x159) [0xca69b9]
 > /path/to/sozu/target/debug/sozu(sozu_lib::protocol::kawa_h1::Http<Front,L>::connect_to_backend+0xb7a) [0xcaafba]
 > /path/to/sozu/target/debug/sozu(sozu_lib::protocol::kawa_h1::Http<Front,L>::ready_inner+0x423) [0xcaf3a3]
 > /path/to/sozu/target/debug/sozu(<sozu_lib::protocol::kawa_h1::Http<Front,L> as sozu_lib::protocol::SessionState>::ready+0x2d) [0xcb040d]
 > /path/to/sozu/target/debug/sozu(<sozu_lib::http::HttpStateMachine as sozu_lib::protocol::SessionState>::ready+0xf2) [0xdbb9b2]
 > /path/to/sozu/target/debug/sozu(<sozu_lib::http::HttpSession as sozu_lib::ProxySession>::ready+0x114) [0xdb4b54]
 > /path/to/sozu/target/debug/sozu(sozu_lib::server::Server::ready+0x746) [0xc95846]
 > /path/to/sozu/target/debug/sozu(sozu_lib::server::Server::run+0x66d) [0xc8869d]
 > /path/to/sozu/target/debug/sozu(sozu::worker::begin_worker_process+0x3060) [0x32e480]
 > /path/to/sozu/target/debug/sozu(sozu::main+0x42e) [0x556afe]
 > /path/to/sozu/target/debug/sozu(core::ops::function::FnOnce::call_once+0xb) [0x1e453b]
 > /path/to/sozu/target/debug/sozu(std::sys_common::backtrace::__rust_begin_short_backtrace+0xe) [0x287e6e]
 > /path/to/sozu/target/debug/sozu(std::rt::lang_start::{{closure}}+0x11) [0x2ff2e1]
 > /path/to/sozu/target/debug/sozu(std::rt::lang_start_internal+0x42e) [0x18217fe]
 > /path/to/sozu/target/debug/sozu(std::rt::lang_start+0x3a) [0x2ff2ba]
 > /path/to/sozu/target/debug/sozu(main+0x1e) [0x556f6e]
 > /usr/lib/libc.so.6(__libc_init_first+0x90) [0x27cd0]
 > /usr/lib/libc.so.6(__libc_start_main+0x8a) [0x27d8a]
 > /path/to/sozu/target/debug/sozu(_start+0x25) [0x17bca5]