Speed up parsing: branch-free exponent, UTF-16 steps, clang parity

[Algunenano]

Parsing gets faster on every compiler I tested (GCC 11-16 and clang 17-22), with two layout-dependent exceptions on one CPU explained below. There are three themes: fewer branch mispredictions on exponents, tweaking the source so clang generates code similar to GCC's, and faster UTF-16 digit parsing. Changes were judged by the benchmark on every compiler: using only instruction and spill counts turned out to be poor predictors.

Please note that the changes can be reviewed per commit.

Exponents without mispredictions: The sign of the exponent is as unpredictable as the input, and for out-of-range values it also decides between underflow and overflow. Three places branched on it; now none do. All three are needed: removing only the first moves the misprediction to the other two for float, where most of these values go out of range. Found with perf record -b on exponent-heavy input, where the exponent sign test was the most mispredicted branch in the parser.

• Exponent sign: parsed without a branch. It uses | rather than ||: GCC turns the || into a branch on "has a sign", which is the same bit when positive exponents are written without +.
• compute_float: both ends of the range are checked with one unsigned comparison, and zero vs infinity is selected without a branch.
• Out-of-range test in from_chars_float_advanced: still one branch marked unlikely, as in Fold the format at compile time under clang, mark cold checks unlikely #413, but tested with one comparison (the exponent is zero or infinite), and the error is selected inside it. Clang otherwise splits the condition into one jump per case.

Clang on par with GCC: Inspired by #413: I compared the code both compilers generate and changed the source where clang did clearly worse. Neither commit changes GCC's code.

• The parser is inlined into from_chars under clang (fastfloat_clang_really_inline, as in Fold the format at compile time under clang, mark cold checks unlikely #413). Clang kept it as a call; GCC already inlines it.
• compute_float's range check is skipped under clang when the exponent passed Clinger's range test. GCC derives this by itself. The zero mantissa check stays (a zero mantissa can get there when the rounding mode is not to nearest).

Leaner digit loops, and faster UTF-16:

• One subtraction per digit in the integer part, the fraction tail and the exponent, for both the test and the value. Both compilers computed the digit twice in some of them. The exponent's overflow guard moves out of the loop (only needed past 18 digits): inside, both compilers turned it into a cmov per digit on the path to the power of ten.
• UTF-16: a 4-unit step after the 8-unit loop, as char already has, so a remaining 4-7 digit run skips the serial scalar loop.
• UTF-16: digit pairs are combined in the vector unit (pmaddwd by 10,1 then 100,1; NEON vmulq/vpaddlq), on the characters, with the '0' offset taken off i * 10^8, which is ready early. parse_eight_digits_unrolled on the packed bytes needs three 64-bit constants in general registers.

Register pressure under GCC (raised in #414): I could reproduce it with GCC 14 and 15 only, not 13 or 16. In the UTF-16 path, GCC spilled values around the 8-unit loop, and with the fraction change also the value inside parse_eight_digits_unrolled, on the mantissa chain. The vector pair combining removes those spills. The same spill remains once per value in the char path on GCC 15: every alternative I found costs clang (see the discarded list).

Measurements: AMD Ryzen 9 7950X3D

Codegen changes between compiler versions and stalls differ between CPUs, even of the same architecture, so these numbers are specific to this setup:

• CPU: AMD Ryzen 9 7950X3D (Zen 4), pinned to core 7 (the CCD with 3D V-Cache, 96 MB L3).
• Compilers: GCC 13.5.0, 14.4.0 and 15.3.0 (official gcc Docker images) and the current ones from the distro (Arch): GCC 16.2.1 (20260810) and clang 22.1.8.
• Building with -O3 -DNDEBUG -static, no -march.
• benchmarks/benchmark.cpp from main (with Average branch misses over the repetitions in the benchmark #415), one run per build (it repeats 1000 times internally). canada and mesh are the bundled files. HIGGS is real data in NumPy's default %.18e format: the first 200k values of the UCI HIGGS dataset (first 100k rows, one value per line). Every value has an exponent, with e-01 and e+00 about evenly split. I've uploaded a copy of the exact dataset I used to https://pastila.nl/?cafebabe/85969be244622b169374bb7a15e768a2#Aun4ig1uL6K4xv4RkNJ3hg==

Cycles per float, main → this PR:

	GCC 13.5	GCC 14.4	GCC 15.3	GCC 16.2	clang 22.1
canada ASCII 64-bit	1.06x	0.99x	0.99x	1.05x	1.19x
canada ASCII 32-bit	1.00x	1.00x	1.04x	1.03x	1.17x
canada UTF-16 64-bit	1.33x	1.25x	1.20x	1.25x	1.41x
canada UTF-16 32-bit	1.19x	1.25x	1.22x	1.26x	1.40x
mesh ASCII 64-bit	1.08x	1.03x	1.02x	1.06x	1.20x
mesh ASCII 32-bit	1.05x	1.05x	1.06x	1.03x	1.32x
mesh UTF-16 64-bit	1.21x	1.15x	1.14x	1.14x	1.25x
mesh UTF-16 32-bit	1.17x	1.11x	1.15x	1.17x	1.37x
HIGGS ASCII 64-bit	1.14x	1.12x	1.18x	1.15x	1.18x
HIGGS ASCII 32-bit	1.08x	1.11x	1.14x	1.12x	1.17x
HIGGS UTF-16 64-bit	1.19x	1.20x	1.15x	1.23x	1.29x
HIGGS UTF-16 32-bit	1.13x	1.15x	1.13x	1.21x	1.28x

Cycles per float

	GCC 13.5	GCC 14.4	GCC 15.3	GCC 16.2	clang 22.1
canada ASCII 64-bit	44.7 → 42.2	42.4 → 43.0	42.6 → 42.9	45.0 → 43.0	51.0 → 42.7
canada ASCII 32-bit	43.1 → 43.0	42.6 → 42.4	45.2 → 43.6	44.7 → 43.4	52.4 → 44.8
canada UTF-16 64-bit	52.5 → 39.4	51.6 → 41.1	50.1 → 41.6	51.9 → 41.5	58.0 → 41.1
canada UTF-16 32-bit	48.2 → 40.4	51.8 → 41.5	51.5 → 42.2	52.5 → 41.6	59.1 → 42.4
mesh ASCII 64-bit	19.2 → 17.8	19.6 → 19.0	19.3 → 18.8	21.2 → 19.9	24.7 → 20.6
mesh ASCII 32-bit	22.4 → 21.4	22.2 → 21.2	23.3 → 21.9	21.9 → 21.3	30.9 → 23.4
mesh UTF-16 64-bit	21.4 → 17.6	21.1 → 18.4	21.0 → 18.4	22.5 → 19.8	26.5 → 21.2
mesh UTF-16 32-bit	23.6 → 20.2	23.9 → 21.5	24.4 → 21.2	25.0 → 21.4	32.6 → 23.8
HIGGS ASCII 64-bit	83.8 → 73.4	82.2 → 73.5	85.0 → 71.7	84.5 → 73.7	89.7 → 76.1
HIGGS ASCII 32-bit	80.6 → 74.8	81.4 → 73.3	84.0 → 73.8	84.2 → 75.4	90.8 → 77.8
HIGGS UTF-16 64-bit	81.7 → 68.7	84.1 → 69.9	80.9 → 70.5	86.7 → 70.5	90.2 → 69.7
HIGGS UTF-16 32-bit	77.4 → 68.7	83.4 → 72.3	80.8 → 71.3	86.1 → 71.4	90.2 → 70.2

Instructions per float go down in all 60 configurations (GCC 3-62, clang 28-76). The two 0.99x cells have 11-12 fewer instructions. Branch misses per float drop on HIGGS from 1.09-1.11 to 0.70-0.71 with every compiler, and do not change elsewhere (within ±0.02).

Clang now runs at 0.90x-1.02x of GCC 16's speed; on main it was 0.71x-0.96x. What remains on canada and mesh is mostly scheduling (same instructions and misses, more memory stalls) and, on mesh, GCC specializing the path where the token ends right after the integer part, which is benchmark-shaped (each value its own string).

Measurements: Intel Xeon 6975P-C

• CPU: Intel Xeon 6975P-C (Granite Rapids), in a KVM virtual machine, pinned to core 7.
• Compilers: GCC 13.5.0, 14.4.0, 15.3.0 and 16.2.0 (official gcc Docker images), and the distro ones (Ubuntu 22.04): GCC 11.4.0 and clang 17.0.6, 18.1.8, 19.1.7, 20.1.8, 21.1.8 and 22.1.8.
• Same build flags, benchmark and data as above, one run per build.

Cycles per float, main → this PR:

	GCC 11.4	GCC 13.5	GCC 14.4	GCC 15.3	GCC 16.2	clang 17.0	clang 18.1	clang 19.1	clang 20.1	clang 21.1	clang 22.1
canada ASCII 64-bit	0.98x	1.09x	1.01x	1.00x	0.87x	1.23x	1.17x	1.23x	1.20x	1.21x	1.19x
canada ASCII 32-bit	1.09x	1.11x	1.06x	1.08x	1.04x	1.21x	1.18x	1.19x	1.20x	1.17x	1.17x
canada UTF-16 64-bit	1.23x	1.29x	1.21x	1.22x	1.23x	1.42x	1.37x	1.39x	1.38x	1.37x	1.38x
canada UTF-16 32-bit	1.28x	1.31x	1.19x	1.09x	1.12x	1.38x	1.32x	1.37x	1.36x	1.37x	1.38x
mesh ASCII 64-bit	1.08x	1.07x	1.02x	1.09x	0.94x	1.28x	1.09x	1.28x	1.26x	1.23x	1.22x
mesh ASCII 32-bit	1.04x	1.05x	1.08x	1.06x	1.04x	1.28x	1.24x	1.27x	1.25x	1.25x	1.23x
mesh UTF-16 64-bit	1.17x	1.18x	1.21x	1.21x	1.11x	1.32x	1.34x	1.30x	1.32x	1.34x	1.29x
mesh UTF-16 32-bit	1.14x	1.16x	1.11x	1.03x	1.01x	1.33x	1.42x	1.30x	1.30x	1.30x	1.28x
HIGGS ASCII 64-bit	1.15x	1.13x	1.16x	1.21x	1.15x	1.08x	1.09x	1.24x	1.24x	1.22x	1.23x
HIGGS ASCII 32-bit	1.14x	1.08x	1.15x	1.19x	1.17x	1.11x	1.08x	1.19x	1.22x	1.22x	1.19x
HIGGS UTF-16 64-bit	1.13x	1.18x	1.18x	1.13x	1.17x	1.17x	1.09x	1.21x	1.20x	1.22x	1.22x
HIGGS UTF-16 32-bit	1.16x	1.14x	1.11x	1.06x	1.09x	1.13x	1.08x	1.20x	1.23x	1.19x	1.20x

Cycles per float

	GCC 11.4	GCC 13.5	GCC 14.4	GCC 15.3	GCC 16.2	clang 17.0	clang 18.1	clang 19.1	clang 20.1	clang 21.1	clang 22.1
canada ASCII 64-bit	36.2 → 37.0	37.1 → 34.2	37.0 → 36.6	36.4 → 36.5	36.0 → 41.3	42.9 → 34.8	42.0 → 35.9	42.5 → 34.7	41.9 → 34.8	42.1 → 34.7	42.4 → 35.7
canada ASCII 32-bit	37.5 → 34.3	38.0 → 34.1	36.6 → 34.5	37.3 → 34.5	36.1 → 34.6	42.9 → 35.3	42.1 → 35.8	42.0 → 35.3	42.3 → 35.4	42.2 → 36.2	43.2 → 37.0
canada UTF-16 64-bit	44.6 → 36.2	44.7 → 34.5	44.9 → 37.2	44.8 → 36.8	44.0 → 35.7	51.1 → 36.0	50.6 → 37.0	49.8 → 35.8	49.2 → 35.5	48.8 → 35.5	49.2 → 35.8
canada UTF-16 32-bit	44.8 → 35.1	44.8 → 34.2	45.0 → 37.7	44.4 → 40.8	44.7 → 39.9	51.8 → 37.5	50.6 → 38.3	49.7 → 36.4	49.3 → 36.3	49.4 → 36.0	50.0 → 36.1
mesh ASCII 64-bit	20.2 → 18.7	17.4 → 16.2	17.2 → 16.8	18.5 → 16.9	17.6 → 18.8	21.7 → 17.0	22.3 → 20.4	22.0 → 17.2	21.7 → 17.3	21.2 → 17.2	22.4 → 18.3
mesh ASCII 32-bit	21.6 → 20.8	19.8 → 18.8	19.8 → 18.4	20.1 → 18.9	19.8 → 19.1	25.6 → 20.0	25.7 → 20.7	25.2 → 19.9	24.9 → 19.9	24.8 → 19.9	25.4 → 20.7
mesh UTF-16 64-bit	20.9 → 17.9	19.4 → 16.5	19.8 → 16.3	19.2 → 15.9	19.4 → 17.4	23.5 → 17.8	25.0 → 18.6	23.1 → 17.8	24.2 → 18.4	23.6 → 17.6	23.8 → 18.4
mesh UTF-16 32-bit	23.2 → 20.3	21.7 → 18.6	22.1 → 19.8	21.9 → 21.3	22.1 → 21.9	27.9 → 20.9	30.3 → 21.2	27.7 → 21.3	26.9 → 20.7	27.2 → 20.9	28.8 → 22.5
HIGGS ASCII 64-bit	82.1 → 71.4	82.8 → 73.6	81.9 → 70.3	86.0 → 70.9	84.8 → 73.9	76.3 → 70.4	78.5 → 71.7	88.9 → 71.8	88.1 → 71.3	87.2 → 71.2	87.5 → 71.4
HIGGS ASCII 32-bit	83.0 → 73.0	82.1 → 75.7	82.0 → 71.1	85.6 → 71.8	85.4 → 72.8	77.0 → 69.4	78.6 → 73.0	87.6 → 73.7	86.9 → 71.5	88.1 → 71.9	86.6 → 72.5
HIGGS UTF-16 64-bit	83.3 → 73.5	84.1 → 71.5	87.4 → 74.3	82.7 → 73.0	85.2 → 72.8	78.8 → 67.4	79.2 → 72.5	88.7 → 73.4	87.3 → 72.6	86.6 → 71.0	87.1 → 71.7
HIGGS UTF-16 32-bit	83.0 → 71.9	82.5 → 72.6	86.6 → 78.3	81.5 → 77.3	84.1 → 77.0	78.6 → 69.3	79.8 → 73.8	88.5 → 73.5	87.7 → 71.1	88.1 → 73.9	88.0 → 73.1

Instructions per float go down in all 132 configurations (GCC 2-63, clang 22-92). Branch misses per float drop on HIGGS from 1.07-1.12 to 0.68-0.75, except with clang 17 and 18, which are already at 0.65-0.69 on main (so their HIGGS gains are smaller). They do not change elsewhere.

With this PR, clang 19-22 are roughly on par with GCC here; on main they were about 15-25% slower on canada and mesh.

The only slowdowns are GCC 16 ASCII 64-bit on canada (0.87x) and mesh (0.94x). Three more runs gave the same (0.88x and 0.96x). This is code layout. Most of it comes from 784cd67 (canada: 37.0 → 41.1 cycles per float), which barely changes the code GCC 16 generates. The extra cycles match the extra slots where the front end delivers no micro-ops (IDQ_UOPS_NOT_DELIVERED.CORE, 11.5G → 16.1G over the whole run), while almost all micro-ops come from the decoded instruction cache in both builds. Rebuilding both commits with -falign-functions=64 -falign-loops=32 flips the result: 37.5 → 34.9, so this PR is 7% faster there. The smaller step at 3e852fe (35.9 → 37.3) also mostly goes away with 64-byte alignment (35.5 → 35.7). GCC 11 canada ASCII 64-bit (0.98x) is neutral over re-runs (36.8 → 37.0-37.2).

Almost all of this work was on x86-64. The NEON versions of the UTF-16 steps are included; I did not tune anything specifically for aarch64, but it was measured on real hardware (below).

Measurements: AWS Graviton4 (Neoverse-V2)

• CPU: AWS Graviton4 (Neoverse-V2, aarch64, 2.7 GHz), c8g.metal-48xl (bare metal), pinned to core 7.
• Compilers: the distro ones (Ubuntu 24.04): GCC 13.3.0 and 14.2.0, and clang 19.1.7, 21.1.8 and 22.1.8 (apt.llvm.org).
• Same build flags, benchmark and data as above. Median of three runs per build; runs of the same build are within 1% of each other (3.7% in one cell of GCC 14 on main).

Cycles per float, main → this PR:

	GCC 13.3	GCC 14.2	clang 19.1	clang 21.1	clang 22.1
canada ASCII 64-bit	1.03x	1.03x	1.33x	1.32x	1.32x
canada ASCII 32-bit	1.04x	1.00x	1.28x	1.31x	1.30x
canada UTF-16 64-bit	1.14x	1.16x	1.32x	1.32x	1.30x
canada UTF-16 32-bit	1.13x	1.12x	1.29x	1.30x	1.32x
mesh ASCII 64-bit	1.04x	1.03x	1.11x	1.11x	1.10x
mesh ASCII 32-bit	1.03x	1.01x	1.25x	1.19x	1.19x
mesh UTF-16 64-bit	1.04x	1.07x	1.23x	1.22x	1.22x
mesh UTF-16 32-bit	1.11x	1.07x	1.27x	1.27x	1.29x
HIGGS ASCII 64-bit	1.14x	1.13x	1.25x	1.25x	1.25x
HIGGS ASCII 32-bit	1.12x	1.12x	1.26x	1.26x	1.26x
HIGGS UTF-16 64-bit	1.07x	1.08x	1.17x	1.17x	1.17x
HIGGS UTF-16 32-bit	1.08x	1.09x	1.18x	1.18x	1.18x

Cycles per float

	GCC 13.3	GCC 14.2	clang 19.1	clang 21.1	clang 22.1
canada ASCII 64-bit	29.0 → 28.1	28.9 → 28.1	36.5 → 27.5	36.4 → 27.6	36.3 → 27.6
canada ASCII 32-bit	29.9 → 28.9	28.5 → 28.5	36.1 → 28.3	37.0 → 28.3	36.8 → 28.3
canada UTF-16 64-bit	34.9 → 30.7	36.6 → 31.6	39.8 → 30.2	40.2 → 30.4	39.3 → 30.3
canada UTF-16 32-bit	35.1 → 31.0	35.5 → 31.7	39.5 → 30.6	40.1 → 30.8	40.5 → 30.6
mesh ASCII 64-bit	13.8 → 13.3	13.7 → 13.3	17.1 → 15.4	17.0 → 15.3	17.0 → 15.5
mesh ASCII 32-bit	15.8 → 15.3	15.1 → 14.9	21.0 → 16.8	20.9 → 17.6	20.9 → 17.6
mesh UTF-16 64-bit	15.3 → 14.7	16.7 → 15.5	19.1 → 15.6	19.2 → 15.8	18.9 → 15.5
mesh UTF-16 32-bit	17.7 → 15.9	17.1 → 16.1	22.6 → 17.8	22.7 → 17.8	23.0 → 17.8
HIGGS ASCII 64-bit	56.2 → 49.2	54.5 → 48.4	61.5 → 49.3	61.6 → 49.1	61.8 → 49.3
HIGGS ASCII 32-bit	56.5 → 50.2	55.1 → 49.2	61.2 → 48.7	61.8 → 49.0	61.7 → 48.9
HIGGS UTF-16 64-bit	55.8 → 51.9	57.5 → 53.3	60.0 → 51.2	59.8 → 51.2	60.0 → 51.3
HIGGS UTF-16 32-bit	56.4 → 52.4	58.3 → 53.5	59.8 → 50.8	60.6 → 51.3	60.2 → 50.9

Instructions per float go down in 58 of 60 configurations (by up to 23 with GCC, 21-67 with clang); the other two are GCC 14 mesh ASCII 64-bit (+0.1) and HIGGS UTF-16 64-bit (+1.3), which are still 1.03x and 1.08x faster. Branch misses per float drop on HIGGS from 1.16-1.50 to 0.68-0.90 with GCC and from 1.05-1.08 to 0.67-0.75 with clang, and do not change elsewhere (within ±0.02). There are no slowdowns; the smallest gain is 1.00x (GCC 14 canada ASCII 32-bit).

With this PR, clang 19-22 run at 0.90x-1.05x of GCC 14's speed except on mesh ASCII (0.85x-0.87x); on main they were at 0.72x-0.98x. The test suite (including the supplemental tests) passes on this machine with GCC 14 and clang 22.

Tried and discarded

• Fraction tail in one masked 8-byte step: same instructions on canada, more on HIGGS.
• Keeping the result out of memory in clang (copies of value at the cold calls): neutral.
• Integer part without end checks when enough characters remain: more instructions for both compilers.
• Peeling the first two exponent digits: HIGGS 1.01x-1.06x faster, but GCC canada 0.95x (GCC reorganizes the whole inlined loop).
• A plain loop for the integer part (from IRainman's fork): 0.86x-1.00x.
• Skipping the rounding-mode probe: about 1% on mesh, not worth a macro.
• clang flags for tail duplication and jump threading: no effect.
• SSE4.2 16-digit step from IRainman's fork (Provide x86-64-v2 optimizations #247): HIGGS ASCII 1.02x-1.10x, but canada and mesh 0.90x-1.02x, and it needs -msse4.2 or higher.
• Branch-free leading sign, like the exponent sign: it resolves right after the first load, so its mispredictions are cheap; branch-free it cost 2-17 instructions per float (0.72x-1.03x).
• Branch-free second product in compute_product_approximation: HIGGS 1.12x-1.28x (branch misses 0.71 → 0.31 per float), but canada 0.89x-0.98x and mesh 32-bit 0.91x-0.97x. A trade I left out; it is a small change if you want it.
• Branch-free last 3 fraction digits: clang 0.69x-0.73x (every step lands on the mantissa chain).
• Serial reduction in parse_eight_digits_unrolled (16- then 32-bit lanes, as in Lemire's "Quickly parsing eight digits"): removes the GCC 15 spill in the char path, but its chain is 3 cycles longer; clang mesh 64-bit 0.95x-0.96x.
• SIMD digit-test constants as arrays in memory: GCC 15 folds them back into mov+movd+pshufd.
• Carrying the exponent sign as an integer mask, or folding it into exponent: 3-12 more instructions per value on GCC 13-16. Its register pressure is GCC keeping the sign character instead of a bool, which is also the cheapest form.
• Rederiving start_digits in the long-mantissa path to free a register: fewer spills, but 0.91x-0.97x on GCC 14 and 16 UTF-16.
• Moving the long-mantissa path out of line: passing the struct by reference keeps it in memory on the hot path (4-21 more instructions); passing scalars was neutral to slightly slower on mesh.
• Changing parsed_number_string_t types (bitfield flags, int32_t exponent): moves spills from one GCC version to another, and it changes the API.

Disclosure

Most of the analysis, benchmarking and coding in this PR was done with an LLM (Claude). I guided it, gave ideas and mostly kept pushing it until I was happy, and I reviewed the code.

[Algunenano]

@lemire I'd say this is ready to review.

I've added a couple of extra commits and run benchmark in more machines (my local 7950X3D, EC2 Intel Xeon 6975P-C and Graviton 4). Out of all the configurations the only slow down appears with the Intel 6975P-C, on GCC 16.2.0 (exactly, 16.2.1 changes the codegen) and it's gone with different layout so in general it positive, both with x86 and ARM, and with clang matching gcc's performance.

I've also included a final section of things I've attempted that didn't work (either my ideas, or ideas posted in issues/forks) in case somebody wants to have a look.