cut: validate UTF-8 once per line and batch writes in -c and -b - #13744
cut: validate UTF-8 once per line and batch writes in -c and -b#13744sylvestre wants to merge 1 commit into
Conversation
Merging this PR will degrade performance by 10.02%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Memory | cut_characters |
20.4 KB | 32.4 KB | -37.03% |
| ❌ | Memory | cut_bytes |
20.5 KB | 32.5 KB | -36.87% |
| ❌ | Simulation | df_with_path |
571.4 µs | 699.3 µs | -18.28% |
| ❌ | Memory | cut_characters_long_lines |
69 KB | 81 KB | -14.82% |
| ❌ | Simulation | cut_bytes |
18.7 ms | 19.5 ms | -4.24% |
| ❌ | Simulation | cksum_multiple_files |
60.6 ms | 63 ms | -3.8% |
| ❌ | Simulation | du_max_depth_balanced_tree[(6, 4, 10)] |
59.8 ms | 62.1 ms | -3.62% |
| ⚡ | Simulation | cut_characters_long_lines |
34.6 ms | 24.1 ms | +43.3% |
| ⚡ | Simulation | cut_characters |
23.3 ms | 21.2 ms | +9.81% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing sylvestre:cut-perf (1b1cfbc) with main (21d4e96)
Footnotes
-
46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
The -c walk decoded one character at a time even on valid UTF-8, and every line cost at least two write_all calls through a dyn Write. Validate the span the walk can touch (four bytes per character wanted, capped at the line) once per line with simdutf8. Within that prefix every non-continuation byte starts a character, so the walk consumes whole spans at once -- n characters occupy at least n bytes -- counting the character starts inside each span with bytecount, which the tree already uses in wc and tr. Everything else -- invalid input, short walks, -b -n, non-UTF-8 locales -- keeps the exact per-character path, whose ASCII-run scan now comes from bstr's find_non_ascii_byte instead of a hand-rolled loop. Selected parts of lines are collected in a buffer flushed at 8 KiB -- enough to batch hundreds of lines per write while staying resident in L1 -- unless stdout is a terminal, where flushing stays per line to keep output interactive. On 40k lines of ~100 characters, a third of them multi-byte, cutting -c 20-70 drops from 28.5 ms to 19.6 ms (GNU cut: 30.1 ms); short mixed lines gain about 10%. Output is byte-identical to the previous implementation on valid, invalid, and truncated UTF-8, including lines made only of stray continuation bytes.
| if let Err(e) = result { | ||
| return Err(USimpleError::new(1, e.to_string())); | ||
| } | ||
| if let Err(e) = out.write_all(&buf) { | ||
| return Err(USimpleError::new(1, e.to_string())); | ||
| } |
There was a problem hiding this comment.
This looks a bit odd, and I would use map_err in both cases. In the first case it also allows you to get rid of the result variable.
| while 0 < cap && cap < line.len() && line[cap] & 0xC0 == 0x80 { | ||
| cap -= 1; | ||
| } | ||
| match simdutf8::basic::from_utf8(&line[..cap]) { |
There was a problem hiding this comment.
The spell checker complains about simdutf here and in other places.
|
yeah, sorry, wip given the perf regressions :) |
2 participants
The -c walk decoded one character at a time even on valid UTF-8, and every line cost at least two write_all calls through a dyn Write.
On 40k lines of ~100 characters, a third of them multi-byte, cutting -c 20-70 drops from 28.5 ms to 19.6 ms (GNU cut: 30.1 ms); short mixed lines gain about 10%. Output is byte-identical to the previous implementation on valid, invalid, and truncated UTF-8, including lines made only of stray continuation bytes.