Why does writing to stdout/stderr from multiple threads results in interleaved output?

[jpmckinney]

What version of the csv crate are you using?

1.2.1

Briefly describe the question, bug or feature request.

Writing to stdout/stderr from multiple threads results in split or mixed lines.

Include a complete program demonstrating a problem.

I'm using rayon. This avoids threading boilerplate. (That said, if the bug is specific to rayon, it's worth looking into, since rayon is a top-100 crate.)

// `extern` to be able to copy-paste into Rust Playground.
extern crate rayon;
use rayon::prelude::*;

fn main() {
    (1..10).par_bridge().for_each(|x| {
        let mut writer = csv::Writer::from_writer(std::io::stderr());
        for _ in 1..1000 {
            writer.serialize((x, "qwertyuiop", "asdfghjkl", "zxcvbnm")).unwrap();
        }
        writer.flush().unwrap();
    });
}

[package]
name = "hello_cargo"
version = "0.1.0"
edition = "2021"

[dependencies]
csv = "1.2"
rayon = "1.0"

What is the observed behavior of the code above?

You'll see lines like:

6,qwertyuiop,asdfghjkl,z9,qwertyuiop,asdfghjkl,zxcvbnm

Later, the rest of the line from the 6 thread will appear (coincidentally mixed with another line from the 9 thread):

9,qwertyxcvbnm

What is the expected or desired behavior of the code above?

Write a full line at once to stdout/stderr. println! and eprintln! manage (I don't know how.)

For example, cannot reproduce with:

// `extern` to be able to copy-paste into Rust Playground.
extern crate rayon;
use rayon::prelude::*;

fn main() {
    (1..10).par_bridge().for_each(|x| {
        for _ in 1..1000 {
            println!("{},{},{},{}", x, "qwertyuiop", "asdfghjkl", "zxcvbnm");
        }
    });
}

Can use this regular expression to find expected long lines: ^.{31}

[BurntSushi]

This is absolutely expected behavior. Your println! code is not equivalent. When you call println!, it's internally calling std::io::stdout().lock(). The csv::Writer does no such thing. All it takes is something that implements std::io::Write. There's no lock() method to call in that context. Instead, it's up to you, the caller, to either pass a locked writer or manage the synchronization yourself.

Basically, you're creating a bunch of threads, each with their own CSV writer, but on top of the same underlying std::io::Write implementation. The CSV writer might call write multiple times on the given writer for each serialize call, and thus the output is interleaved. And it is expected and correct. There is quite literally nothing the csv crate can do for you to prevent it. You have to prevent it.

I can think of two very simple modifications to your code:

1. Create your csv writers on top of an in-memory buffer. Then when the thread exits (or at defined points in time), dump the buffer to stderr with one write call.
2. Do the equivalent of what println! is doing: add a mutex that you acquire before you call serialize and released after serialize returns.

[jpmckinney]

Thanks! The pieces I didn’t know were what println was doing to avoid this (locking) and what csv was willing to guarantee (or not). All makes sense.