Skip to content

+= string append does not re-pair split surrogate (astral/emoji corruption; crashes TUIs on emoji) #6741

Description

@proggeramlug

Summary

+= string append does not re-pair a split UTF-16 surrogate, so astral characters (emoji) built up one code unit at a time are corrupted. Expression concat and Node handle this correctly; perry's += path does not.

Minimal repro

const e = "a👋b";            // real astral char
const hi = e[1], lo = e[2];  // lone high + low surrogate from indexing

const X = hi + lo;               // expression concat
let Y = ""; Y += hi; Y += lo;    // incremental += append

console.log(X.codePointAt(0), [...X].length);  // node & perry: 128075 1
console.log(Y.codePointAt(0), [...Y].length);  // node: 128075 1  |  perry: 55357 2  ← BUG
console.log(X === Y);                           // node: true      |  perry: false

Root cause

js_string_append (crates/perry-runtime/src/string/append.rs) copies operand bytes verbatim and never calls canonicalize_surrogate_pairs, unlike every js_string_concat* path. A lone high surrogate at the end of dest and a lone low surrogate at the start of src are stored as two separate 3-byte WTF-8 sequences instead of the astral code point's 4-byte UTF-8. codePointAt, [...s].length, and Intl.Segmenter grapheme segmentation then all disagree with Node.

Impact

Any string rebuilt code-unit-by-code-unit over emoji/astral text — ANSI stripping, manual parsers, terminal width calculation — is silently corrupted. Found while running the pi coding-agent TUI under perry: pi's visibleWidth strips ANSI via stripped += clean[i], which splits an emoji's surrogate pair across two += appends, over-counting the rendered line width and tripping pi's Rendered line exceeds terminal width invariant → uncaughtException, crashing the UI on any emoji in a colored line (e.g. a normal streamed assistant reply). Discovered while validating #6728.

Fix

PR incoming: propagate the lone-surrogate flag through append and canonicalize when a high→low pair straddles the dest|src join (detected in O(1) from the boundary bytes, so ASCII appends stay allocation-free). Mirrors js_string_concat. Differential regression test included.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions