Summary
+= string append does not re-pair a split UTF-16 surrogate, so astral characters (emoji) built up one code unit at a time are corrupted. Expression concat and Node handle this correctly; perry's += path does not.
Minimal repro
const e = "a👋b"; // real astral char
const hi = e[1], lo = e[2]; // lone high + low surrogate from indexing
const X = hi + lo; // expression concat
let Y = ""; Y += hi; Y += lo; // incremental += append
console.log(X.codePointAt(0), [...X].length); // node & perry: 128075 1
console.log(Y.codePointAt(0), [...Y].length); // node: 128075 1 | perry: 55357 2 ← BUG
console.log(X === Y); // node: true | perry: false
Root cause
js_string_append (crates/perry-runtime/src/string/append.rs) copies operand bytes verbatim and never calls canonicalize_surrogate_pairs, unlike every js_string_concat* path. A lone high surrogate at the end of dest and a lone low surrogate at the start of src are stored as two separate 3-byte WTF-8 sequences instead of the astral code point's 4-byte UTF-8. codePointAt, [...s].length, and Intl.Segmenter grapheme segmentation then all disagree with Node.
Impact
Any string rebuilt code-unit-by-code-unit over emoji/astral text — ANSI stripping, manual parsers, terminal width calculation — is silently corrupted. Found while running the pi coding-agent TUI under perry: pi's visibleWidth strips ANSI via stripped += clean[i], which splits an emoji's surrogate pair across two += appends, over-counting the rendered line width and tripping pi's Rendered line exceeds terminal width invariant → uncaughtException, crashing the UI on any emoji in a colored line (e.g. a normal streamed assistant reply). Discovered while validating #6728.
Fix
PR incoming: propagate the lone-surrogate flag through append and canonicalize when a high→low pair straddles the dest|src join (detected in O(1) from the boundary bytes, so ASCII appends stay allocation-free). Mirrors js_string_concat. Differential regression test included.
Summary
+=string append does not re-pair a split UTF-16 surrogate, so astral characters (emoji) built up one code unit at a time are corrupted. Expression concat and Node handle this correctly; perry's+=path does not.Minimal repro
Root cause
js_string_append(crates/perry-runtime/src/string/append.rs) copies operand bytes verbatim and never callscanonicalize_surrogate_pairs, unlike everyjs_string_concat*path. A lone high surrogate at the end ofdestand a lone low surrogate at the start ofsrcare stored as two separate 3-byte WTF-8 sequences instead of the astral code point's 4-byte UTF-8.codePointAt,[...s].length, andIntl.Segmentergrapheme segmentation then all disagree with Node.Impact
Any string rebuilt code-unit-by-code-unit over emoji/astral text — ANSI stripping, manual parsers, terminal width calculation — is silently corrupted. Found while running the pi coding-agent TUI under perry: pi's
visibleWidthstrips ANSI viastripped += clean[i], which splits an emoji's surrogate pair across two+=appends, over-counting the rendered line width and tripping pi'sRendered line exceeds terminal widthinvariant →uncaughtException, crashing the UI on any emoji in a colored line (e.g. a normal streamed assistant reply). Discovered while validating #6728.Fix
PR incoming: propagate the lone-surrogate flag through append and canonicalize when a high→low pair straddles the
dest|srcjoin (detected in O(1) from the boundary bytes, so ASCII appends stay allocation-free). Mirrorsjs_string_concat. Differential regression test included.