"error: failed to download <dependency>" because of CARGO_HOME

[TriplEight]

Some logs are listed in #4007.

Problem
Happens when there is a custom CARGO_HOME dir (can't really try with the default) and it starts having different versions of the same dependency. In this case, something is being removed in $CARGO_HOME/registry and it causes different errors.

Some examples of errors:

The most popular one:

cargo +nightly build

    Compiling wasm-build-runner-impl v1.0.0 (/builds/parity/substrate/target/wasm32-unknown-unknown/debug/wbuild-runner/node-runtime4102221825354268953)
    Finished dev [unoptimized + debuginfo] target(s) in 15.37s
     Running `/builds/parity/substrate/target/wasm32-unknown-unknown/debug/wbuild-runner/node-runtime4102221825354268953/target/x86_64-unknown-linux-gnu/debug/wasm-build-runner-impl`
thread 'main' panicked at '`cargo metadata` can not fail on project `Cargo.toml`; qed: CargoMetadata { stderr: "error: failed to download `assert_cmd v1.0.1`\n\nCaused by:\n  unable to get packages from source\n\nCaused by:\n  failed to parse manifest at `/ci-cache/substrate/cargo/check-web-wasm/registry/src/github.com-1ecc6299db9ec823/assert_cmd-1.0.1/Cargo.toml`\n\nCaused by:\n  can\'t find `bin_fixture` bin, specify bin.path\n" }', /builds/parity/substrate/utils/wasm-builder/src/wasm_project.rs:95:26

cargo check

Updating crates.io index
    Updating git repository `https://github.com/paritytech/subdb`
    Updating git repository `https://github.com/paritytech/substrate`
error: failed to load source for a dependency on `sp-database`
Caused by:
  Unable to update https://github.com/paritytech/substrate?branch=gav-db-trait#94048157
Caused by:
  revspec '9404815700a840586fc8760a60180f4e1bf97ce4' not found; class=Reference (4); code=NotFound (-3)
Command exited with non-zero status 101
17.51user 1.50system 0:26.14elapsed 72%CPU (0avgtext+0avgdata 278424maxresident)k
112inputs+380880outputs (31major+190933minor)pagefaults 0swaps

cargo deny check

2020-04-15 13:07:52 [ERROR] Error during execution of `cargo metadata`:     Updating crates.io index
    Updating git repository `https://github.com/paritytech/subdb`
    Updating git repository `https://github.com/paritytech/substrate`
error: failed to load source for a dependency on `sp-database`
Caused by:
  Unable to update https://github.com/paritytech/substrate?branch=gav-db-trait#94048157
Caused by:
  revspec '9404815700a840586fc8760a60180f4e1bf97ce4' not found; class=Reference (4); code=NotFound (-3)

cargo +nightly check

error: failed to download `assert_cmd v1.0.1`
Caused by:
  unable to get packages from source
Caused by:
  failed to parse manifest at `/ci-cache/substrate/cargo/cargo-check-benches/registry/src/github.com-1ecc6299db9ec823/assert_cmd-1.0.1/Cargo.toml`
Caused by:
  can't find `bin_fixture` bin, specify bin.path
Command exited with non-zero status 101
0.34user 0.11system 0:00.90elapsed 50%CPU (0avgtext+0avgdata 53604maxresident)k
127768inputs+0outputs (37major+17008minor)pagefaults 0swaps

cargo test --release

    Compiling wasm-build-runner-impl v1.0.0 (/builds/parity/substrate/target/release/wbuild-runner/substrate-test-runtime5390084481911088889)
    Finished release [optimized] target(s) in 1m 40s
     Running `/builds/parity/substrate/target/release/wbuild-runner/substrate-test-runtime5390084481911088889/target/x86_64-unknown-linux-gnu/release/wasm-build-runner-impl`
thread 'main' panicked at '`cargo metadata` can not fail on project `Cargo.toml`; qed: CargoMetadata { stderr: "    Updating crates.io index\nwarning: spurious network error (2 tries remaining): failed to create locked file \'/ci-cache/substrate/cargo/test-linux-stable/registry/index/github.com-1ecc6299db9ec823/.git/objects/pack/pack_git2_GTidx.lock\': No such file or directory; class=Os (2)\nwarning: spurious network error (1 tries remaining): failed to create temporary file \'/ci-cache/substrate/cargo/test-linux-stable/registry/index/github.com-1ecc6299db9ec823/.git/objects/pack/pack_git2_NucRGO\': No such file or directory; class=Os (2)\nerror: failed to fetch `https://github.com/rust-lang/crates.io-index`\n\nCaused by:\n  failed to create temporary file \'/ci-cache/substrate/cargo/test-linux-stable/registry/index/github.com-1ecc6299db9ec823/.git/objects/pack/pack_git2_8MHXKD\': No such file or directory; class=Os (2)\n" }', /builds/parity/substrate/utils/wasm-builder/src/wasm_project.rs:95:26

They are always reproducible and do not depend on the network. Purging CARGO_HOME always helps.

Steps

@ehuss

I'm pretty unfamiliar with gitlab CI, and particularly how it handles caching.

This repro is very close to how our cache in GitLab is organised:

1. download the corrupted cache: https://send.firefox.com/download/24dbf35a0d54b6cf/#dRuJUk3XEbh99Dpd08lyCw
2. unarchive it in a 'tar xf -C /place/you/want'
3. sudo docker run -it -v /place/you/want/:/cache/ parity/rust-builder:latest
4. inside container:

    export CARGO_HOME=/cache/cargo-check-benches/
    git clone https://github.com/paritytech/substrate .
    BUILD_DUMMY_WASM_BINARY=1 time cargo +nightly check --benches --all

5. and you will get the error

Possible Solution(s)

As a workaround, cargo vendor could help, but didn't try it so far.

It would be really cool if cargo respected CIs, instead of locking files.

Notes

The output of cargo version: started to be a problem in the middle of stable 1.41, and still stands.

This happens in CI where we have the shared CARGO_HOME (and I've already excluded the locking problem, it doesn't cause this kind of errors, the worst case with it is a parallel job waits). CI runs code within docker containers and the cache dirs is mapped via docker volumes, exactly as in repro. Docker containers are based on debian:buster.

@ehuss

You might want to check that the filesystem supports the style of locking Cargo uses (flock with LOCK_EX on unix).

We have ext4 everywhere.

[TriplEight]

And I owe you a short explanation of why it's a problem.

We used to have this setup with shared CARGO_CACHE for around a year and it worked flawlessly. And we didn't change anything with its setup.

Basically, our cache was structured like this:

/build-host/cicache/<project>/<cargo-cache>/<ci-job-name> and every on all the branches shared that CARGO_HOME there, via docker volume mapping.
From time to time, on pretty active projects, file locks happened, but there never were any errors, a job just waited for a sec. And we even had "the cruel cleanup script" on the build-hosts, it was removing all files older than 5 days everywhere, including CARGO_HOME/registry

After we started to stumble upon this bug,

1. I've thought that it might happen because of the "the cruel cleanup script", and fixed it to not to go into CARGO_HOME/registry since cargo and git do some cleanup there.
2. Thought it's because of the data races and tried separating shared cache dir, and made it like this:
   /build-host/cicache/<project>/<cargo-cache>/<branch-name>/<ci-job-name> so every branch had it's own cache. But it didn't help.
   Apparently, the bug may happen in the same branch if you have one dependency with different versions.
3. Now it happens so often so I had to remove CARGO_HOME caching from CI entirely.

[lroux-at-jellysmack]

Hello, still happens in nightly-2022-03-01, the weird thing is we didn't have any problem until just today !

[nrdxp]

I just hit this as well, was hitting a very strange issue in a docker build that didn't happen on a local build where cargo vendor returned "no such file or directory" when trying to open the Cargo.toml of atty v0.2.14. However I confirmed the file existed before cargo was executed, so it must have somehow deleted it during execution. I confirmed it was the same file using strace.

If I unsert CARGO_HOME beforehand, then it finishes successfully.