Skip to content

perf(sftp): size the tar decision from listings, not a stat per file (#494) - #564

Merged
kipavy merged 1 commit into
devfrom
fix/tar-probe-fallback-494
Oct 7, 2026
Merged

kipavy merged 1 commit into
devfrom
fix/tar-probe-fallback-494

Conversation

@kipavy

@kipavy kipavy commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Closes #494.

When the exec size probe (find … -size / Get-ChildItem) can't answer (no probe for the shell, a failure, or the 120 s timeout), has_large_remote walked the tree itself to find a file ≥ 64 MiB. That walk duplicated walk()'s traversal and stat'ed every entry: one round trip per file plus two per directory. SFTP listings already carry each entry's type and size.

The issue offered "go per-file directly". I didn't do that: it would send every small-file tree on a probe-less host (including Windows OpenSSH's default cmd.exe) per file instead of tar, which is a slowdown. This keeps every routing decision as it was and makes the fallback cheaper.

Change

  • resume/mod.rs: walk()'s traversal becomes visit(), which takes a callback returning Go / Prune / Stop. walk() is rebuilt on it with the same output: same order, same symlink skipping, the same unsafe-name skipping, and the same report_skipped for unreadable metadata.
  • resume/large.rs: the fallback is now large_by_walk() on visit(). On SFTP it costs one round trip per directory, none per file, and stops at the first large file. For a tree of 10,000 files in 200 folders, that's about 200 round trips instead of about 10,400.
  • Listed.complete: whether the listing vouches for the entry's type and size. large_by_walk stats anything it doesn't vouch for, exactly as before:
    • SFTP: complete only when the server sent permissions, plus a size for non-directories (listed_completely).
    • Local disk: always complete.
    • Docker, FTP, WebDAV (From<RemoteFile>): never complete, since e.g. docker's listing writes stat || echo 0 and can't tell an unknown size from 0. Those endpoints keep the old per-entry lookups.
    • Unreadable metadata (stat: None): also stat'ed individually, as before.

Tests

  • New: a large file is found at any depth; only the selected items are stat'ed and each folder is listed once ((1, 5) for a 4-folder tree); the walk stops at the first large file ((1, 1)); a listing that doesn't vouch is stat'ed entry by entry and still finds the large file; listed_completely for each attribute gap.
  • TestFs counts stats and lists, and can serve bare listings.
  • Full cargo test --lib against this PR's base (69ddb1bd), in the same container: base 789 passed, this branch 794 passed (the 5 new tests). Both fail the same 3 commands::plugins tests, because the JS plugin bundles weren't built in that container. cargo fmt --check and cargo clippy --all-targets -D warnings are clean.

Not measured

  • A timing run against a real server. The speed-up is shown by the stat and list counts in the tests.

🤖 Generated with Claude Code

…494)

When the exec size probe can't answer, has_large_remote walked the tree
with its own traversal and stat'ed every entry: one round trip per file
plus two per directory. SFTP listings already carry each entry's type
and size.

walk()'s traversal becomes visit(), with an early stop; walk() and the
new large_by_walk() both use it. On SFTP the fallback now costs one
round trip per directory and stops at the first large file.

Listed gains `complete`: whether the listing vouches for the entry's
type and size. Entries it doesn't vouch for (a bare SFTP listing, and
every docker/FTP/WebDAV listing, where an unknown size reads as 0) are
stat'ed exactly as before, so no endpoint routes differently.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kipavy
kipavy merged commit 8cfebe4 into dev Oct 7, 2026
4 checks passed
@kipavy
kipavy deleted the fix/tar-probe-fallback-494 branch October 7, 2026 10:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant