New .cargo format forces me to remove .cargo to run `cargo generate-lockfile` in sources/

[mikn]

I recently tried to upgrade to Twoliter 0.8.0 where you changed the format of how you vendor dependencies into the .cargo folder to make it more robust against version changes. Inherently great! But sadly if you've built a kit or a variant that has resulted in a .cargo folder this gets picked up by cargo (I am unsure how to disable it):
https://internals.rust-lang.org/t/problems-of-cargo-config-files-and-possible-solutions/12987/14
rust-lang/cargo#12738

And for me it creates a recursive loop that gives an error like this:

mikn@hal8999 ~/d/m/s/p/b/b/sources ((72abfc6e))> cargo generate-lockfile
error: failed to get `bottlerocket-defaults-helper` as a dependency of package `settings-defaults-metal-dev v0.3.0 (/home/mikn/devel/molnett/src/platform/base/bottlerocket/sources/settings-defaults/metal-dev)`

Caused by:
  failed to load source for dependency `bottlerocket-defaults-helper`

Caused by:
  Unable to update https://github.com/bottlerocket-os/bottlerocket-settings-sdk?tag=bottlerocket-defaults-helper-v0.1.1

Caused by:
  the source git+https://github.com/bottlerocket-os/bottlerocket-settings-sdk?tag=bottlerocket-defaults-helper-v0.1.1 requires a lock file to be present first before it can be
  used against vendored source code

  remove the source replacement configuration, generate a lock file, and then
  restore the source replacement configuration to continue the build
mikn@hal8999 ~/d/m/s/p/b/b/sources ((72abfc6e)) [101]> 

I can fix this by removing the .cargo folder, but I do need to do that every time which feels a bit crufty.

The secondary problem is that it is also a lot more sensitive to races, as now it seems to expect to be allowed to do more in this folder than it was previously. For now I am downgrading to 0.7.3 again.

[jmt-lab]

Hi there Mikael sorry that this changed caused some headache for you. I think what is happening is a conflict with cargo vendor's relative paths. I believe since you are in sources that its going up seeing the root .cargo/config.toml and then erroring on finding the vendor which might lead to some weirdness. I'm doing some investigatory work right now.

[jmt-lab]

Ah I see we aren't checking in Cargo.lock with our bottlerocket-settings-sdk package which is kind of a bad idea here and leading to this pain point.

[jmt-lab]

So i have a fix and getting a PR up for it to make our internal use of cargo vendor not clash with external uses of cargo for our case. We are planning a 0.8.1 release this week and so hopefully we can include it there.

[mikn]

Oh really, quick find!

If you are interested, the race I am encountering is this:

[2025-03-12T20:41:42Z DEBUG twoliter__bin::common] Running: Command { std: "cargo" "make" "--disable-check-for-updates" "--makefile" "/home/builder/.bazel/execroot/_main/platform/base/bottlerocket/build/tools/Makefile.toml" "--cwd" "/home/builder/.bazel/execroot/_main/platform/base/bottlerocket" "-e=TLPRIVATE_SDK_IMAGE=public.ecr.aws/bottlerocket/bottlerocket-sdk:v0.50.1" "-e=BUILDSYS_OUTPUT_GENERATION_ID=1" "-e=TWOLITER_TOOLS_DIR=/home/builder/.bazel/execroot/_main/platform/base/bottlerocket/build/tools" "-e=BUILDSYS_ARCH=x86_64" "-e=BUILDSYS_VARIANT=metal-dev" "-e=BUILDSYS_VERSION_IMAGE=1.2.0" "-e=GO_MODULES=" "-e=BUILDSYS_UPSTREAM_SOURCE_FALLBACK=false" "build", kill_on_drop: false }
2025-03-12 20:41:48 [ERROR] failed to fetch crates:     Blocking waiting for file lock on package cache
    Updating git repository `https://github.com/bottlerocket-os/bottlerocket-settings-sdk`
warning: spurious network error (3 tries remaining): failed to resolve address for github.com: Temporary failure in name resolution; class=Net (12)
warning: spurious network error (2 tries remaining): failed to resolve address for github.com: Temporary failure in name resolution; class=Net (12)
warning: spurious network error (1 tries remaining): failed to resolve address for github.com: Temporary failure in name resolution; class=Net (12)
error: failed to get `bottlerocket-defaults-helper` as a dependency of package `settings-defaults-metal-dev v0.3.0 (/tmp/sources/settings-defaults/metal-dev)`

Caused by:
  failed to load source for dependency `bottlerocket-defaults-helper`

Caused by:
  Unable to update https://github.com/bottlerocket-os/bottlerocket-settings-sdk?tag=bottlerocket-defaults-helper-v0.1.1#0cf85735

Caused by:
  failed to fetch into: /tmp/.cargo/git/db/bottlerocket-settings-sdk-3ffd24dec00e96c2

Caused by:
  network failure seems to have happened
  if a proxy or similar is necessary `net.git-fetch-with-cli` may help here
  https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli

Caused by:
  failed to resolve address for github.com: Temporary failure in name resolution; class=Net (12)

    Blocking waiting for file lock on package cache
   Compiling metal-dev v0.1.0 (/home/builder/actions-runner/_work/src/src/platform/base/bottlerocket/variants/metal-dev)

As alluded in other PRs and comments, I am wrapping twoliter in Bazel and in the variant build step I launch several variant builds (out of the same folder). This worked before the change to vendoring into .cargo - do you have any ideas on how to address it? I am a bit stumped as I think this is a result of the format of the .cargo folder with the vendor folder vs how it was before and the error seems to be entirely internal to cargo.

[bcressey]

Ah I see we aren't checking in Cargo.lock with our bottlerocket-settings-sdk package which is kind of a bad idea here and leading to this pain point.

This is pretty typical for library-only crates like the settings SDK. Not sure why it's a bad idea, can you say more?

[jmt-lab]

Eh its not really a bad idea, i misspoke here, i got myself confused on the loop error and misread, i thought cargo was complaining about their not being lockfiles with the git dependencies :P

[jmt-lab]

As alluded in other PRs and comments, I am wrapping twoliter in Bazel and in the variant build step I launch several variant builds (out of the same folder). This worked before the change to vendoring into .cargo - do you have any ideas on how to address it? I am a bit stumped as I think this is a result of the format of the .cargo folder with the vendor folder vs how it was before and the error seems to be entirely internal to cargo.

Hmm, yeah so before we were copying the entire .cargo folder and that included the .git .registry folders, this is where we got into trouble with rust compatibility. I'm a bit confused why its not respecting the vendor folder when running in parallel on the same folder. I'm guessing that the parallel runs are clobbering cargo vendor folder since we always run cargo vendor each time.

[jmt-lab]

Hey mikn, just as a heads up we have release twoliter v0.8.1 which should address this issue, feel free to give it a try and report back if you run into any issues.