Skip to content

fix(watcher): stop ignored batches and small Flag::Rescan markers from breaking the index (#690)#693

Open
gustav-fff wants to merge 1 commit into
mainfrom
triage-bot/issue-690
Open

fix(watcher): stop ignored batches and small Flag::Rescan markers from breaking the index (#690)#693
gustav-fff wants to merge 1 commit into
mainfrom
triage-bot/issue-690

Conversation

@gustav-fff

Copy link
Copy Markdown
Collaborator

Closes #690

Root cause

Two independent bugs in handle_debounced_events (crates/fff-core/src/watcher/background_watcher.rs):

  1. affected_paths_count summed raw debounced_event.event.paths.len() at background_watcher.rs:457 before ignore filtering. A burst of >1024 events under an ignored subtree (e.g. target/ churn from rustdoc / cargo build on macOS FSEvents recursive stream) forced a full rescan even though every path was filtered out at background_watcher.rs:451-453.
  2. The paths.len() < 16 && all(!is_dir && !is_ignored) bypass at background_watcher.rs:365-374 silently breaked out of the event loop on Flag::Rescan without scheduling a rescan and without processing later events in the batch. On Linux, inotify Q_OVERFLOW arrives with no paths at all, so all() was vacuously true and every overflow marker was dropped on the floor. On macOS, any batch of <16 small non-ignored paths hit the same bypass.

Fix

  • Count only actionable paths (those actually pushed to paths_to_remove / dirs_to_remove / paths_to_add_or_modify) against MAX_OVERFLOW_FILES. Ignored churn no longer trips the guard.
  • Always schedule a full rescan on need_rescan(). Log event.info() alongside the paths so user dropped vs kernel dropped is distinguishable.

Architectural fix #2 from the issue (adaptive vs unconditional recursive FSEvents strategy) is intentionally out of scope — it needs benchmarking and a public strategy knob.

Steps to reproduce

Trigger 1 (ignored batch → false rescan):

```sh

Any recursively watched macOS repo with .gitignore containing /target

cargo doc --workspace # or any rustdoc generator producing many .js under target/doc

fff-search emits: "Too many affected paths in a single batch, triggering full rescan"

affected_paths_count = 1025, exactly matching the > 1024 branch,

even though every path is under /target and filtered out of the index.

```

Trigger 2 (silent Flag::Rescan bypass): unit-testable directly. On pre-fix main:

```sh
cd crates/fff-core

Feed a batch containing a single non-ignored, non-dir path with Flag::Rescan set.

Expected: watch_registry dispatches a Rescan event.

Actual: handler breaks out early; subscribers never see the rescan.

```

Both triggers are covered by unit tests added in this PR:

  • ignored_batch_does_not_trigger_full_rescan — 2000 Create(File) events under .gitignored target/ must not dispatch a Rescan.
  • rescan_flag_still_triggers_rescan_when_batch_is_small — a single-path Flag::Rescan must dispatch a Rescan.
  • empty_path_rescan_flag_triggers_rescan — the Linux Q_OVERFLOW shape (empty path list) must dispatch a Rescan.

All three tests fail on origin/main (bug 1 dispatches a spurious rescan; bugs 2/3 silently return with no dispatch).

How verified

```sh
cargo test -p fff-search --lib background_watcher

5 passed; 0 failed — includes 3 new regression tests.

cargo test -p fff-search --lib

127 passed; 0 failed.

cargo test -p fff-search --test watch_subscription_test

11 passed; 0 failed.

cargo test -p fff-search --test new_directory_watcher_test

5 passed; 0 failed.

cargo fmt --package fff-search --check

clean

```

Automated triage via Gustav. Honk-Honk 🪿

…m breaking the index (#690)

Two independent correctness bugs in handle_debounced_events:

1. affected_paths_count summed raw event.paths.len() before ignore
   filtering. A burst of >1024 events under an ignored subtree (e.g.
   target/ churn from rustdoc/cargo build) forced a full rescan even
   though every event was filtered out. Now only actionable paths
   (non-ignored, non-.git, added to the remove/add/dir buckets) count
   toward the guard.

2. The `debounced_event.event.paths.len() < 16 && all(!is_dir &&
   !is_ignored)` bypass for Flag::Rescan silently returned without
   scheduling a rescan and without processing subsequent events in
   the batch. On Linux, inotify Q_OVERFLOW arrives with no paths at
   all, making the `all()` vacuously true and dropping every overflow
   marker on the floor. Always schedule a full rescan on need_rescan()
   and log event.info() so `user dropped` vs `kernel dropped` is
   distinguishable.

Refs #690
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: tons of full rescans while tracked repo files are barely changed

1 participant