consult-buffer slow in directories with many entries due to hidden project sources

[hpfr]

consult-buffer takes several seconds to show candidates when invoked in buffers with default-directory somewhere under /gnu/store/ on my system. ls | wc -l reports 227,466 entries in this directory, which seems relevant. The behavior occurs even in a subdirectory like /gnu/store/13rjyfjhh4w69zjd1lv1xman302s93bm-emacs-29.4/, which only has a few entries.

The delay went away when removing the three consult--source-project sources. Based on the profiler, it seems locate-dominating-file called by project-current called by consult--project-root is the culprit. This makes sense, project-current is looking for .git or similar in /gnu/store/ when I’m in a subdirectory of any depth. (benchmark-run 10 (project-current) in /gnu/store/ gives 3 seconds with 4 garbage collections taking 0.57 seconds. The same benchmark in my ~/repos/emacs/lisp/gnus (i.e. parent directories with a few hundred children) takes a couple milliseconds or less.

I find myself in /gnu/store/ often because I install Emacs and its packages via Guix, so visiting any definitions takes me there. Of course I can just remove the hidden project sources from consult-buffer and use the dedicated consult-project-buffer when I’m actually in projects. I’m opening the issue for SEO for future searchers and am perfectly happy with simply closing it as a known limitation, but if anyone can think of a general solution, I’m all ears. You could also drop the hidden project sources if you think avoiding this corner case is more important than having them available from consult-buffer, but since I think I’m the first to report it, it’s probably a pretty niche issue.

[minad]

The problem here is that Consult assumes that project-current is very cheap. (Recently Emacs got a project-mode-line which makes the same assumption, even worse project-current is executed during redisplay!)

Unfortunately project.el has its limitations and the assumption doesn't really hold. I always wondered why performance of all the project functions is not a higher priority, in particular determining the project root. One could imagine some caching mechanism. This is also what I suggest for your use case. Use a custom project root function which bails out quickly in guix directories or which implements its own caching. I also use my own project root function which bails out in slow remote directories.

I should also add some Guix criticism - you wrote that the store has ~200k entries. This seems like a problematic design. Not that the kernel or the fs cannot handle this, but there is simply too much in user space which cannot. You could maybe report Emacs bugs if you think that locate-dominating-file (or other Emacs APIs) should be more efficient in the presence of so many files?

[hpfr]

Yeah, I agree that upstream needs to consider the performance characteristics of project-current and locate-dominating-file. I’ll see about a bug. I bet TRAMP and network filesystems cause similar issues.

In your case I suggest to use a custom project root function which bails out quickly in guix directories or which implements its own caching.

True, ignoring /{nix,gnu}/store specifically is not pretty but an easy workaround. Thank you!

(defvar my/huge-directories '("/gnu/store/" "/nix/store/")
  "List of directories known to have huge numbers of entries.
Some Elisp is written under the assumption that browsing the
filesystem is fast enough that blocking the UI isn’t a concern
for a non-critical operation. This variable is used in
workarounds for these cases.")

(defun my/file-in-huge-directory-p (file)
  "Return non-nil if FILE is somewhere in ‘my/huge-directories’."
  (seq-some (apply-partially #'file-in-directory-p file) my/huge-directories))

(defun my/file-not-in-huge-directory-p (file)
  "Return non-nil if FILE is not in ‘my/huge-directories’."
  (not (my/file-in-huge-directory-p file)))

;; HACK: Return nil early when interactive functions try to check for a project
;; in directories which can contain so many entries that
;; ‘locate-dominating-file’ is slow enough to impact performance.
(advice-add 'project--find-in-directory :before-while #'my/file-not-in-huge-directory-p)

I should also add some Guix criticism - you wrote that the store has ~200k entries. This seems like a problematic design. Not that the kernel or the fs cannot handle this, but there is simply too much in user space which cannot.

Meh, I think “user space” is too broad. Command line shells and terminal emulators are perfectly fine with it. ls /gnu/store in Ghostty takes two seconds. find /gnu/store -maxdepth 1 takes even less. Piping to head cuts it down to under half a second. Piping to less makes interacting with the output trivial. So this is too slow to happen as a background operation of some rich/smart interface (as is the case with Emacs checking for a project), but fast enough that doing real work in the directory is not a problem from a CLI. Nautilus takes far too long to be useful but at least doesn’t block the UI and shows a spinner, so it’s easy to go somewhere else. So I do think it’s more an issue with Emacs assuming locate-dominating-file is a cheap operation for synchronous use, as you already pointed out.

Doing the sharding thing .git/objects does wouldn’t hurt, though. I think this would break binary substitutes though so it probably can’t happen. I digress!

[minad]

I bet TRAMP and network filesystems cause similar issues.

Indeed. For this reason my custom project root function ignores remote mounts and tramp paths. I had thought before about adding some consult-project-exclude-regexp, but then I think users should rather write their own project root function or the issue should be fixed in project.el in the first place. Maybe projectile does better here? One should also not forget that users may use projects located on tramp paths.

Also as I mentioned, I wonder about project-mode-line. This must perform badly if you are inside a tramp directory. I really hope for some improvements here.

Meh, I think “user space” is too broad. Command line shells and terminal emulators are perfectly fine with it. ls /gnu/store in Ghostty takes two seconds. find /gnu/store -maxdepth 1 takes even less. Piping to head cuts it down to under half a second. Piping to less makes interacting with the output trivial. So this is too slow to happen as a background operation of some rich/smart interface (as is the case with Emacs checking for a project), but fast enough that doing real work in the directory is not a problem from a CLI. Nautilus takes far too long to be useful but at least doesn’t block the UI and shows a spinner, so it’s easy to go somewhere else. So I do think it’s more an issue with Emacs assuming locate-dominating-file is a cheap operation for synchronous use, as you already pointed out.

I find two seconds pretty slow for the terminal. The file names only should be a few megabytes of data? Nautilus doesn't work. Emacs doesn't work. Reasons enough to reconsider. For a GNU system you have to consider the whole picture. You are on Hurd I presume? ;) Of course any fs operation on the terminal will still be O(logn) fast as long as you don't perform any readdir operations which go over all directory entries.

Doing the sharding thing .git/objects does wouldn’t hurt, though. I think this would break binary substitutes though so it probably can’t happen. I digress!

Exactly. Git and other databases use such an approach for a reason.