feat: add directory-level CoW copy for faster worktree creation#163
Open
ryoppippi wants to merge 2 commits into
Open
feat: add directory-level CoW copy for faster worktree creation#163ryoppippi wants to merge 2 commits into
ryoppippi wants to merge 2 commits into
Conversation
Add directory-level copy support that dramatically speeds up copying large directories like node_modules when creating new worktrees. Platform-specific optimisations: - macOS (APFS): Use clonefile(2) on entire directories for instant CoW clones, falling back to recursive file-by-file clonefile on failure. - Linux: Add FICLONE ioctl support for CoW reflink on Btrfs/XFS, falling back to io.Copy (which uses copy_file_range internally). - Other platforms: Recursive walk with io.Copy. The copy logic now groups files by top-level directory and attempts a single directory-level copy when all files within that directory are being copied (no NoCopy exclusions). This avoids the overhead of listing and copying hundreds of thousands of individual files. Benchmark results on macOS (Apple M3 Pro) with a real 5.9GB node_modules (265K files): copyDir (clonefile): ~21s, 3.6 KB alloc file-by-file: ~85s, 543 MB alloc ~4x faster with 150,000x less memory allocation.
- Add nolint:gosec annotations for internal copy functions (G703, G115) - Use 0600 permissions in test WriteFile calls (G306) - Fix LICENCE -> LICENSE spelling (misspell US locale)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
copyDir) that copies entire directories in one operation instead of file-by-fileclonefile(2)on directories for instant APFS CoW clones, with fallback to recursive file-by-file copyFICLONEioctl (viaunix.IoctlFileClone) for CoW reflink on Btrfs/XFS, with fallback toio.Copy(which usescopy_file_rangeinternally)CopyFilesToWorktreenow groups files by top-level directory and attempts directory-level copy when all files in that directory are being copied (no NoCopy exclusions apply)Benchmark
On macOS (Apple M3 Pro, APFS) with a real 5.9GB
node_modules(265K files, 40K directories):copyDir(clonefile)copyFile~4x faster, 150,000x less memory.
With a synthetic 2,500-file directory: ~12x faster (~31ms vs ~383ms).
Test plan
TestCopyDir,TestCopyDir_IndependentOfSource,TestCopyFilesToWorktree_DirectoryLevelCopy,TestCopyFilesToWorktree_DirectoryLevelCopy_WithNoCopyBenchmarkCopyDir_Small,BenchmarkCopyFileByFile_Small,BenchmarkCopyDir_RealNodeModules,BenchmarkCopyFileByFile_RealNodeModules(setBENCH_NODE_MODULESenv var)