Description
While dealing with a performance library in requests (yes, the Python library) I stumbled across psf/requests#2371. I wanted to quickly evaluate whether switching to Rust would make sense for my problem so I created a small benchmark in Rust, matching the ones in https://github.com/alex/http-client-bench:
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let client = Client::new();
let uri = "http://localhost:8080".parse()?;
let mut resp = client.get(uri).await?;
let mut handle = stdout();
while let Some(chunk) = resp.body_mut().data().await {
handle.write_all(&chunk?).await?;
}
Ok(())
}
To my surprise this is a lot slower than the Python clients on my machine:
$ ./run.sh
Python 3.7.4
go version go1.13.8 darwin/amd64
BENCH HTTPLIB:
8.00GiB 0:00:25 [ 317MiB/s] [================================>] 100%
BENCH URLLIB3:
8.00GiB 0:00:33 [ 244MiB/s] [================================>] 100%
BENCH REQUESTS
8.00GiB 0:00:36 [ 222MiB/s] [================================>] 100%
BENCH GO HTTP
8.00GiB 0:00:23 [ 351MiB/s] [================================>] 100%
signal: broken pipe
BENCH RUST HYPER
8.00GiB 0:00:59 [ 136MiB/s] [================================>] 100%
Error: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }
I've build it with --release, timings vary between runs but it's always the same ballpark. I thought at first that maybe it's just writing to stdout that's slow but even if I modify the benchmark to remove writing to stdout it gets faster but remains slower than the competition:
extern crate hyper;
use hyper::Client;
use hyper::body::HttpBody as _;
// use tokio::io::{stdout, AsyncWriteExt as _};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let client = Client::new();
let uri = "http://localhost:8080".parse()?;
let mut resp = client.get(uri).await?;
// let mut handle = stdout();
let bench_size: usize = 8 * 1024 * 1024 * 1024;
let mut bytes_seen: usize = 0;
while let Some(chunk) = resp.body_mut().data().await {
bytes_seen += chunk?.len();
if bytes_seen >= bench_size { break; }
// handle.write_all(&chunk?).await?;
}
Ok(())
}
$ time ./target/release/http-client-bench
real 0m47.529s
user 0m31.867s
sys 0m34.785s
I've tried search the documentation for any configuration changes I might be able to make but nothing looked relevant as far as I could see. So... what's going on?