Provide path feature with git resource

[yanganto]

Problem

We want to use a customized git dependency of a crate in a repository, which have more than one Rust crates and the workspace in not in the root of repository.

Following is a real example from WasmEdge

.
├── bindings
│   └── rust
│       ├── Cargo.toml
│       ├── wasmedge-sdk
│       │   └── Cargo.toml
│       └── wasmedge-sys
│           └── Cargo.toml

166 directories, 717 files

If we can use git and path at the same time, everything will be straight forward for developers.
wasmedge-sys = { git = "wasmedge_git_url", path = "/bindings/rust/wasmedge-sys"}

You can check out the following closed issue, people still there and request this feature.
#1462

Proposed Solution

We need to modify the following structure, and allow using path with git, also the path will be transformed to absolute path before using, such that we can make sure the path will not go out of the resource.

src/cargo/core/source/source_id.rs

  #[derive(Eq, Clone, Debug)]
  struct SourceIdInner {
      /// The source URL.
      url: Url,
      /// The canonical version of the above url
      canonical_url: CanonicalUrl,
      /// The source kind.
      kind: SourceKind,
      /// For example, the exact Git revision of the specified branch for a Git Source.
      precise: Option<String>,
      /// Name of the registry source for alternative registries
      /// WARNING: this is not always set for alt-registries when the name is
      /// not known.
      name: Option<String>,
      /// The portion of resource
+     /// For example, the folder for a Git source
+     portion: Option<String>,
  }

src/cargo/sources/config.rs

  struct SourceConfigDef {
      /// Indicates this source should be replaced with another of the given name.
      replace_with: OptValue<String>,
      /// A directory source.
      directory: Option<ConfigRelativePath>,
      /// A registry source. Value is a URL.
      registry: OptValue<String>,
      /// A local registry source.
      local_registry: Option<ConfigRelativePath>,
      /// A git source. Value is a URL.
      git: OptValue<String>,
      /// The git branch.
      branch: OptValue<String>,
      /// The git tag.
      tag: OptValue<String>,
      /// The git revision.
      rev: OptValue<String>,
+     /// The porton of the Source,
+     /// for example the folder of git resoutce
+     portion: Option<String>,
  }

Notes

No response

[ehuss]

Can you say more about why you would like to specify the path within the git repository? Cargo already searches for the package by name, so unless you have two packages with the exact same name, this should not be necessary.

[yanganto]

Thanks ehuss. It works.

[zapkub]

@ehuss what about the repository contain more than two packages with the same name?

[dantliff-sqc]

@ehuss:

Can you say more about why you would like to specify the path within the git repository? Cargo already searches for the package by name, so unless you have two packages with the exact same name, this should not be necessary.

I'm pretty sure you've answered your own question here :)

Also, in large mono-repositories, this search has a non-trivial cost.

And for repositories that contain multiple cargo workspaces, the crate name is not name-spaced by the workspace path, so naming conflicts are more likely. E.g. two workspaces in the same repository that both contain a (different) crate called "utils" or "ci".

[weihanglo]

For performance-wise discussion, see #14395.

For package name conflicts in a git dependency, see #11858.

[epage]

@dantliff-sqc one of the risks associated with #14395 (improving performance) is that we won't get enough benefit because most packages in the workspace will be loaded anyways. If you have a concrete example of only a fraction of the packages being loaded (once dependencies have been fully walked), it would help if you could post that there with details of what kind of numbers you are looking at and maybe even a profiling sample (by default, this only includes function calls and not data passed to functions so I can't see this accidentally disclosing details of private code but it might be good to double check the file). This could help us better understand the relative importance of making a change to this.

[DavidAntliff]

@epage sorry, I don't have that data currently. I'm significantly more concerned with name conflicts, although to date I've avoided this across all workspaces in the mono repo I work on, using C-style prefixes (e.g. bar-utils, foo-utils).

When you say "will be loaded anyways" what do you mean exactly? Loaded by whom? I do have a use-case where a third-party consumes the mono-repo via git in order to just include a small handful of crates across the workspaces, and it's certainly not building most of them.

[epage]

When you say "will be loaded anyways" what do you mean exactly? Loaded by whom? I do have a use-case where a third-party consumes the mono-repo via git in order to just include a small handful of crates across the workspaces, and it's certainly not building most of them.

The cases I've had to work with load most or all of the packages in the repo (directly or indirectly), reducing the benefit. If we have examples of cases where a package with a git dependency only needs a small number of packages (directly or indirectly) from the repo, that would be great to provide details about in the performance issue.

[DavidAntliff]

@epage Ok, I understand. I can only offer my anecdotal experience above, as the repo I am working with now is most definitely not open-source. Perhaps the more common case is as you describe.