Skip to content

fix(hook): convert shell-native cursor unit to byte at the boundary (closes #6) - #11

Merged
ShortArrow merged 1 commit into
developfrom
fix/issue-6-multibyte-cursor
May 31, 2026
Merged

fix(hook): convert shell-native cursor unit to byte at the boundary (closes #6)#11
ShortArrow merged 1 commit into
developfrom
fix/issue-6-multibyte-cursor

Conversation

@ShortArrow

Copy link
Copy Markdown
Owner

Summary

  • Bug (BUG: Insertion of whitespace characters in strange positions when Entering multibyte characters args #6): Shells pass --cursor in their own native unit — bash / zsh / clink / nu count Unicode scalar values (= chars), pwsh counts UTF-16 code units (.NET / PSReadLine native). The Rust hook used the raw value as a byte offset, which agreed by accident on ASCII but split mid-character on Japanese, emoji, or any non-ASCII input.
  • Reporter's PoC (verified on real WSL bash): typing ls ./おはよう ./test1, moving the cursor between and the space, pressing Space repeatedly grew a misplaced gap between and instead of accumulating after . Same misalignment silently corrupted any expansion that ran against a multi-byte buffer.
  • Fix: cmd::hook converts the incoming cursor to a byte offset exactly once at the cmd/app boundary via app::hook::shell_cursor_to_byte (dispatches char_cursor_to_byte for bash/zsh/clink/nu, utf16_cursor_to_byte for pwsh). app::hook::render wraps the domain renderer and converts back on the way out. zsh skips output conversion because its renderer splits LBUFFER/RBUFFER by byte and the cursor number never reaches zsh — converting it would be wrong.
  • pwsh + emoji: utf16_cursor_to_byte handles surrogate pairs (🎯 = 1 char / 4 bytes / 2 UTF-16 code units). Mid-surrogate cursor rounds down to the previous boundary so the slice stays UTF-8 valid.
  • Bake-mode bash dispatcher (0.1.17, Git Bash) unaffected: it slices ${READLINE_LINE:0:READLINE_POINT} in pure bash, which is char-based, so the bake path was correct from the start.
  • Tidy First (same PR per Plan): cmd::hook's three short-circuit paths (oversize line / paste-pending / config-load failure) shared a five-line "insert space at cursor" shape. They now call a shared app::hook::insert_space_action(line, byte_cursor) helper.

Test plan

  • 28 new unit tests in app::hook — four conversion helpers × ASCII / Japanese (3-byte BMP) / emoji (4-byte / 2 UTF-16) / combining / RTL / NUL / empty / past-end clamping / round-trip property against inverses
  • 7 new e2e tests in tests/cli_integration.rsrunex hook --shell <s> for bash / zsh / pwsh / clink / nu against a multi-byte buffer, asserting the shell-correct cursor unit comes back. bash arm is the reporter's PoC verbatim. pwsh arm exercises the emoji surrogate-pair path.
  • Full Windows local cargo test: 685 pass
  • Manual verify on real WSL bash with 0.1.18: reporter's PoC no longer reproduces. Spaces accumulate only between and ./test1, never between and .
  • CI: GitHub Actions Linux / Windows / macOS on this PR
  • Manual smoke after merge on Git Bash + MSYS2 bake path (should be unaffected, but worth confirming the bake path still passes its own gitbash_smoke tests)

Critical files

  • runex/src/app/hook.rs — four conversion helpers + shell_cursor_to_byte dispatcher + insert_space_action helper + render wrapper + 28 unit tests
  • runex/src/cmd/hook.rs — entry conversion at the cmd/app boundary + 3 short-circuit collapse onto the new helper
  • runex/src/domain/hook.rsHookAction doc updated to point at the new app-layer conversion (no behavior change in the domain)
  • runex/src/main.rs--cursor CLI arg doc updated to spell out the per-shell input unit
  • runex/tests/cli_integration.rs — 7 multi-byte e2e tests for runex hook
  • CHANGELOG.md — 0.1.18 entry under Fixed / Internal / Tests
  • runex/Cargo.toml — 0.1.17 → 0.1.18

Closes #6.

…loses #6)

Shells send `--cursor` in their own unit: bash, zsh, clink, and nu
count Unicode scalar values (= chars); pwsh counts UTF-16 code units
(its PSReadLine / .NET native unit). The domain hook treated the
value as a byte offset, which agreed by accident on ASCII but split
mid-character on Japanese, emoji, or any other non-ASCII buffer.

Reporter's PoC reproduced on real WSL bash: typing
`ls ./おはよう ./test1`, moving the cursor between `う` and the
space, then pressing Space repeatedly grew a misplaced gap between
`は` and `よ` instead of accumulating after `う`. Same misalignment
silently corrupted any expansion that ran against a multi-byte
buffer in any shell.

Conversion is done once at the cmd/app boundary by
`app::hook::shell_cursor_to_byte`, which dispatches to
`char_cursor_to_byte` (bash/zsh/clink/nu) or `utf16_cursor_to_byte`
(pwsh). The inverse `byte_cursor_to_char` / `byte_cursor_to_utf16`
runs on the way out inside `app::hook::render`, which wraps the
domain renderer so its per-shell `format!` arms can stay
byte-cursor-only. zsh is excluded from the output conversion
because its renderer emits LBUFFER/RBUFFER substrings produced by
a byte `split_at` — the cursor number itself never crosses the
shell boundary there.

The pwsh path handles the surrogate-pair case (emoji = 1 char / 4
bytes / 2 UTF-16 code units), rounding down to the previous
boundary on a mid-surrogate cursor so the downstream slice stays
UTF-8 valid.

Bake-mode bash dispatcher (0.1.17, Git Bash) is unaffected — it
slices `${READLINE_LINE:0:READLINE_POINT}` in pure bash, which is
char-based, so the bake path was correct from the start.

Tidy First: the three near-identical "insert space at cursor"
short-circuits in cmd::hook::handle (oversize line, paste pending,
config load failure) collapse into one helper
`app::hook::insert_space_action`. Same PR keeps the cleanup local
to the bug fix.

Tests:

- 28 new unit tests in `app::hook` covering the four conversion
  helpers across ASCII / Japanese (3-byte BMP) / emoji (4-byte
  outside BMP / 2 UTF-16 code units) / combining marks / RTL / NUL
  / empty / past-end clamping / round-trip symmetry against
  inverses.
- 7 new e2e tests in `tests/cli_integration.rs::hook_*` driving
  `runex hook --shell <s>` for each of bash / zsh / pwsh / clink /
  nu with a multi-byte buffer and asserting the shell-correct
  cursor unit comes back. bash arm is the reporter's PoC verbatim;
  pwsh arm exercises the emoji surrogate-pair path.
- Manual verify on real WSL bash with 0.1.18: reporter's PoC no
  longer reproduces. Spaces now accumulate only between `う` and
  `./test1`, never between `は` and `よ`.

Bump version 0.1.17 → 0.1.18 + Cargo.lock + CHANGELOG.
@ShortArrow
ShortArrow merged commit 7323375 into develop May 31, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant