Windows support: fix daemon boot (SIGHUP), IPC listener, and ambient hooks; plus reliability fixes - #108
Windows support: fix daemon boot (SIGHUP), IPC listener, and ambient hooks; plus reliability fixes#108Kmmille2 wants to merge 15 commits into
Conversation
The four Claude/Codex capture+recall hooks hardcoded /usr/bin/python3, absent on Windows (Git Bash), so ambient capture silently no-opped via its fail-safe exit 0. Add a self-contained resolver that honors IAI_MCP_PYTHON, skips the Windows Store python3 App-Execution-Alias stub, and falls back to the py launcher. Verified on Windows 11 / Python 3.11.9. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_install_boot_signal_trace built (SIGTERM, SIGINT, SIGHUP); accessing the POSIX-only SIGHUP attribute raised AttributeError before the per-signal try/except could catch it, killing daemon boot on Windows. Resolve each signal lazily via getattr and drop absent ones, matching the run-loop shutdown handler's existing guard. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SocketServer.serve() read inspect.signature(asyncio.start_unix_server) before the IS_WINDOWS branch. That attribute does not exist on Windows, so serve() raised AttributeError immediately; as a create_task'd coroutine the exception was never retrieved, so the daemon ran but its IPC listener never bound (no .daemon.port; 'daemon status' reported not running; MCP hosts could not reach it). Move the POSIX-only signature read past the Windows early return. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…'t kill it A Task-Scheduler-launched daemon attached to the interactive session's console was terminated with STATUS_CONTROL_C_EXIT when the host CLI restarted: the console CTRL_C delivered SIGINT and tripped the shutdown path on a healthy daemon. Install a Windows console-ctrl handler (only when IAI_MCP_LAUNCHD_MANAGED) that swallows CTRL_C/CTRL_BREAK; Task Scheduler still stops it via TerminateProcess. Guarded and wrapped so it can never affect boot elsewhere. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A record capture_turn rejects (e.g. an embedder failure) raised out of both drain loops, so the spool file was never unlinked and every subsequent drain re-hit the same poison record — silently wedging all future capture. Guard the per-record capture_turn call: on any failure, count it skipped and continue so the file still unlinks and the backlog drains. Re-runs stay loss-free via the source_uuid idem-tag. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
recycle() drains idle slots then re-puts them; a concurrent _release() can refill the queue in between, so a re-put can hit queue.Full. That branch closed the slot but never decremented _filled (unlike _release), so the pool believed it held a live slot it didn't. Over a daemon lifetime the accumulated leak drains the pool until borrow()'s get() blocks forever and recall hangs. Decrement _filled under _fill_lock, matching _release(). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
read_overflow_chain reserved Vec::with_capacity(total_len) from a caller- supplied on-disk length; a corrupted/forged varint could make that enormous and hit the allocator abort path (uncatchable, kills the host). Cap the reservation at the physical maximum the chain can provide. cursor.rs payload_of and read_full_cell indexed page[start..start+len] raw from the same unvalidated decode, panicking across the PyO3 boundary on a damaged inline cell; use checked_add + .get().ok_or(Integrity) so a bad page yields a typed error, as raw_leaf_cell_bytes already does. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
6209035 to
284fb05
Compare
|
Really nice work, Kyle, and it cleared our security review. We'll take the Windows fixes and merge them under your name. The three engine hardening commits are good too. They'll come in through our storage-core pipeline rather than this PR, since we keep a single writer on that code, with your authorship on them. The drain poison-pill guard is the one we can't take as-is: skip-and-unlink turns a stuck backlog into silent data loss, and a systemic failure like a cold embedder or a full disk would quietly wipe the whole backlog. If you want to rework it, quarantine the failing event to a Two small asks so I can merge the Windows part: rebase on latest main (3.0.2 just shipped and fixed the snapshot path your commits touch), and drop the bench-CSV churn the |
|
Really appreciate how thorough this is — you got the daemon booting on Windows end-to-end. Two things now that 3.0.5 landed: (1) it reworked the daemon internals, so this now conflicts and needs a rebase; (2) it overlaps #126, which is more surgical and ships tests for the same boot path, so I'm planning to base the Windows daemon-boot fix on that. Would you be up for narrowing this to the unique reliability fixes #126 doesn't cover? Separately, the Rust engine changes ( |
…che) Three Windows-only failures kept every hook capture from ever reaching the store, even though each hook exited 0 (fail-safe): - Turn-capture hook and `capture-turn-deferred` both fsync the state directory after publishing the offset. Windows cannot open a directory with os.open() (PermissionError: [Errno 13]) and has no directory fsync, so the embedded script aborted right after the spool write. Guard the directory fsync with `os.name != "nt"`; same guard in lillibrain `fsync_directory`. - The hook's spool lock used `fcntl.flock`; `fcntl` does not exist on Windows (ModuleNotFoundError). Fall back to `msvcrt.locking(LK_NBLCK)`, which has the same non-blocking semantics. - The Stop hook resolves the CLI via `~/.iai-mcp/.cli-path` before a PATH scan; nothing seeded that cache outside `cowork install`, and the seeder only recognised `iai-mcp`, not `iai-mcp.exe`. A venv/pipx install whose Scripts dir is not on the hook's PATH logged "iai-mcp CLI not found" and exited before rotating the live spool, so nothing ever drained. Seed the cache from `capture-hooks install` for every host target and accept the `.exe` / interpreter-sibling forms. Verified on Windows 11: turn hook rc=0 with no traceback, Stop hook rotates `.live.jsonl`, daemon startup drain consumed the rotated spool (store 488 -> 721 records) and `memory_recall` returns the captured turn. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Thank you so much! This is really valuable. Can you please rebase so I can merge? |
…nstall `capture-hooks install` registered `bash <WindowsPath>` unquoted, e.g. `bash C:\Users\First Last\.claude\hooks\iai-mcp-turn-capture.sh`. The host hands that string to a POSIX shell, which eats the backslashes and splits on the space, so Claude Code logged a non-blocking hook error on every UserPromptSubmit/Stop (`bash: C:UsersKyle: No such file or directory`) and Codex never ran the hook either. Any home directory with a space (`/Users/First Last`) fails the same way on macOS. Register `bash "<posix path>"` via a shared `_hookcmd.hook_command()` for the Claude Code, Codex and Cursor installers, and make reinstall repair an existing registration whose command differs (the marker check previously left the broken form in place forever). Verified: the registered string, run through `sh -c` as the hosts do, exits 0 and logs the turn; both hosts' configs now carry the quoted form. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…in budget
Hook registration: a bare `bash` on Windows resolves, for any host that is
not itself running inside Git Bash (Codex CLI/Desktop, GUI apps), to
`C:\Windows\System32\bash.exe` — the WSL shim — because Git for Windows
only puts `Git\cmd` on the machine PATH. WSL cannot open a `C:/...` script,
so every hook died with `/bin/bash: ...: No such file or directory` (127)
and Codex captured nothing. `hook_command()` now pins the Git Bash
executable (`Git/bin/bash.exe`, x86 and per-user installs, else a PATH hit
that is not the System32/Store shim) and quotes it; POSIX hosts keep `bash`.
Verified with a machine+user-only PATH: before rc=127, after rc=0.
deferred-drain: the per-batch wall-clock budget (120 s + 0.2 s/event, cap
1 h) is now env-overridable (IAI_MCP_DRAIN_BATCH_TIMEOUT_{BASE,PER_EVENT,CAP}_S).
A real backlog on a CPU embedder runs at ~2-3 events/s, so the default
budget kills a healthy child as "wedged" and aborts the whole recovery run;
with the override and 2 MB batches an 88k-event backlog drains steadily.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Follow-up to the Git Bash pin: `"C:/Program Files/Git/bin/bash.exe" "<script>"` still failed under Codex on Windows. A command line that starts with a quote and contains further quotes loses its first and last quote when cmd.exe processes it (the /C rule), so it ran as `C:/Program` and Codex reported every iai-mcp hook as Failed while plugin hooks completed. Register the interpreter and the script as 8.3 short names (GetShortPathNameW) instead — no spaces, no quotes, so the line survives cmd, PowerShell and a direct CreateProcess. Long-path fallback (quoted) when 8.3 names are unavailable. Verified with `codex exec`: SessionStart, UserPromptSubmit and Stop hooks all Completed; the session-recall, turn-capture and session-capture hooks logged the Codex session id and a spool file was written. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
GetShortPathNameW shortens every component, including the hook script's filename (IAI-MC~1.SH), which defeats the installers' marker match (the next install appends a duplicate) and makes the Codex hook browser unreadable. Replace only components that contain a space with their 8.3 name; keep the rest long: `C:/PROGRA~1/Git/bin/bash.exe C:/Users/KYLEMI~1/.codex/hooks/iai-mcp-turn-capture.sh`. Verified with `codex exec`: SessionStart, UserPromptSubmit and Stop hooks Completed and logged the session id; reinstall is idempotent again. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Hey Kyle — any luck with the rebase? The branch still shows conflicts against main, so I can't merge it yet. Happy to help if you're stuck. Once it's rebased on top of current main I'll get it in. |
…rain Three reasons captures never reached the store on a busy Windows machine: - idle_detector had no Windows backend, so os_idle_time_sec() returned nothing and the daemon judged idleness only by the MCP wrapper heartbeat, which stays fresh while any host is open. It never reached DROWSY, so it never drained the capture queue or consolidated. Add a GetLastInputInfo backend (interactive session input idle, the HIDIdleTime counterpart) and report it in status()/describe(). - The startup drain is one-shot: when the background gate is busy at boot (graph preload / community detection over a large store) it returned "deferring to the next cycle", but nothing re-scheduled it. Retry on a cadence (IAI_MCP_BOOT_DRAIN_RETRY_SEC, default 120 s, up to IAI_MCP_BOOT_DRAIN_MAX_ATTEMPTS). - Same-day recall needs a drain while a host is still open. When OS input idle shows the human away for IAI_MCP_WAKE_IDLE_DRAIN_SEC (default 300 s) and the FSM is WAKE, run the drowsy-edge drain, at most once per IAI_MCP_WAKE_IDLE_DRAIN_INTERVAL_SEC (default 600 s). 0 disables. Verified: doctor reports "GetLastInputInfo: 1s PASS"; after a restart with a 135-file queue the retrying boot drain emptied it and turns captured that day were recallable. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
save_state() writes a temp file and os.replace()s it over lifecycle_state.json. On Windows a concurrent reader (hook or CLI load_state, backup copy) holding the target open makes the rename fail at once with PermissionError (WinError 5); the sleep pipeline caught that as a pipeline exception and aborted the whole consolidation cycle. Retry the rename briefly (8 attempts, 50 ms linear backoff) on nt only. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Summary
The daemon doesn't currently start on Windows — two Python-portability bugs kill it before it can serve, so the store stays empty and MCP hosts can't reach it. This PR fixes those, makes the ambient-capture hooks work under Git Bash, and includes three reliability fixes I hit while getting it running.
Verified end-to-end on Windows 11 / Python 3.11 (from-source build, MSVC toolchain): the daemon boots, binds its IPC listener, and
capture → recallreturns records through the MCP path in both Claude Code and Codex. Backfilled ~480 records and confirmed recall + theiai braindashboard.Windows portability
signal.SIGHUPis absent on Windows._install_boot_signal_tracebuilt the tuple(SIGTERM, SIGINT, SIGHUP); evaluatingsignal.SIGHUPraisedAttributeErrorbefore the per-signaltry/exceptcould catch it, killing boot. Resolve each signal lazily viagetattr— mirrors the run-loop shutdown handler's existing guard.asyncio.start_unix_serveris absent on Windows.SocketServer.serve()readinspect.signature(asyncio.start_unix_server)before theIS_WINDOWSbranch; on Windows that attribute doesn't exist, soserve()raised immediately and — as acreate_task'd coroutine — the exception was never retrieved. The daemon ran but was unreachable (no.daemon.port;daemon statusreported not running). Moved the POSIX-only signature read past the Windows early return./usr/bin/python3, which doesn't exist under Git Bash, so capture silently no-opped (fail-safeexit 0). Resolve the interpreter portably: honorIAI_MCP_PYTHON, skip the Windows Storepython3App-Execution-Alias stub, fall back to thepylauncher.*.shto LF via.gitattributesso the hooks run under bash on Windows (autocrlf otherwise materializes CRLF → bad interpreter).CTRL_Csurvival. A Task-Scheduler daemon attached to the interactive console was terminated withSTATUS_CONTROL_C_EXITwhen the host CLI restarted. Install a Windows console-ctrl handler (only underIAI_MCP_LAUNCHD_MANAGED) that swallows CTRL_C/CTRL_BREAK; Task Scheduler still stops it viaTerminateProcess.Reliability (general, not Windows-specific)
capture_turnrejects (e.g. an embedder failure) raised out of both drain loops, so the spool file never unlinked and every subsequent drain re-hit the same record — silently wedging all future capture. Quarantine-skip the failing record so the file still unlinks and the backlog drains (re-runs stay loss-free via thesource_uuididem-tag).recycle(). Theexcept queue.Fullbranch closed a slot without decrementing_filled(unlike_release); a race with a concurrent_release()leaks a slot, and over a daemon lifetime the pool drains untilborrow()'sget()blocks forever. Decrement under_fill_lock, matching_release().lillibrain.read_overflow_chainreservedVec::with_capacity(total_len)from a caller-supplied on-disk length — a corrupted/forged varint hits the allocator abort path (uncatchable, kills the host).cursor.rspayload_of/read_full_cellslicedpage[start..start+len]raw from the same unvalidated decode (panics across PyO3). Cap the reservation at the physical maximum; usechecked_add+.get().ok_or(Integrity)so a damaged page yields a typed error, asraw_leaf_cell_bytesalready does.Happy to split this into two PRs (Windows portability vs reliability) if you'd prefer smaller reviews. Thanks for the project — it's a genuinely nice piece of engineering.