Loading of a git source is slow due to manifest parsing

[epage]

When you have some git dependencies or patches, loading them takes a long time

For the traces from #14238, notice how slow the first resolve is compared to the second. Almost all of that is taken up in Manifest parsing

Note: this slow down is most relevant for tooling like rust-analyzer or rust-defined completions when it may be called in interactive contexts without any compilation happening.

Related: #13724, #14603

[epage]

Previously discussed at https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/Redundant.20code.20in.20.60GitSouce.60.3F

[epage]

The core problem is that git sources are indexed by name. To find a name, we walk the source to find all Cargo.toml files except if they are in a submodule or a hidden directory except if a Cargo.toml has a path dependency in it in an ignored location.

Without that second exception, we could parse Cargo.toml to toml::Table, look up the name, and be done. That would cut out probably 60% of the manifest parsing performance hit.

With the second exception, we have to do a full parse of the Cargo.toml file.

Ideas

• Allow dependencies to specify the path within the repo (Git dependency with a path into a sub-folder location #11858)
  • Needs design work
  • Requires people to opt-in to get performance gain
• Store the path in Cargo.lock
  • Also helps with ensuring consistent path to ambiguous name
  • Potential overlapping design work with Git dependency with a path into a sub-folder location #11858 (source id, package id, package id spec)
  • Investigation is needed whether we can get the information from Cargo.lock to the source to be use
• Store a cache in .cargo-ok
  • Content in the file shouldn't cause checkout thrashing
  • Being in that file keeps the checkout self-contained (delete it and everything is gone) and avoids fingerprinting issues with old cargo (they ignore this file)
  • Likely should only have one entry (thrashing on version changes) as a simple reclamation policy
  • Either fingerprint cargo like we do rustc or use the git commit sha if its not dirty, not caching when it is dirty
• Tell people to avoid ambiguous names, or test files (only reduces part of the problem)
• A file in the repo that indexes it (require opt-in by the dependency)
• A .cargoignmore file in the repo for what not to index (requires opt-in by the dependency)
• Speed up parsing by caching immutable Cargo.tomls in a faster format (likely cut 1/6 of parse time out)

[epage]

Main risk: there aren't enough manifests for us to skip loading

[epage]

Store the path in Cargo.lock

If we are able to do this, we might be able to extend it to also work with recursive paths which might unblock us supporting that (#13360).