flowey: add --use-local-deps option to build-igvm, add local request variant to download_openvmm_deps #2388
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR adds support for using locally-downloaded dependencies in the flowey build system as a step toward reproducible builds. The main change renames download_openvmm_deps to resolve_openvmm_deps and adds a LocalPath request variant alongside the existing Version variant. A new --use-local-deps CLI option is added to the build-igvm pipeline for local testing.
Key changes:
- Renamed
download_openvmm_depsmodule toresolve_openvmm_depsto reflect its dual purpose (download or resolve local paths) - Added
LocalPathrequest variant with logic to resolve dependencies from local x64/aarch64 subdirectories - Added
--use-local-depsand--custom-openvmm-depsCLI options to build-igvm pipeline for testing with local dependencies
Reviewed Changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| flowey/flowey_lib_hvlite/src/resolve_openvmm_deps.rs | Adds LocalPath request variant and implementation to resolve dependencies from local filesystem paths with x64/aarch64 subdirectory structure |
| flowey/flowey_lib_hvlite/src/lib.rs | Updates module declaration from download_openvmm_deps to resolve_openvmm_deps |
| flowey/flowey_lib_hvlite/src/init_vmm_tests_env.rs | Updates import and request references to use resolve_openvmm_deps |
| flowey/flowey_lib_hvlite/src/init_openvmm_magicpath_openhcl_sysroot.rs | Updates import and request references to use resolve_openvmm_deps |
| flowey/flowey_lib_hvlite/src/init_openvmm_magicpath_linux_test_kernel.rs | Updates import and request references to use resolve_openvmm_deps |
| flowey/flowey_lib_hvlite/src/build_openhcl_initrd.rs | Updates import and request references to use resolve_openvmm_deps |
| flowey/flowey_lib_hvlite/src/build_openhcl_igvm_from_recipe.rs | Updates import and request references to use resolve_openvmm_deps |
| flowey/flowey_lib_hvlite/src/_jobs/cfg_versions.rs | Adds LocalDependencyRequest struct and Local/Download request variants; routes local dependency requests to resolve_openvmm_deps |
| flowey/flowey_lib_hvlite/src/_jobs/cfg_common.rs | Updates import to use resolve_openvmm_deps |
| flowey/flowey_hvlite/src/pipelines/vmm_tests.rs | Changes cfg_versions request from struct to Download enum variant |
| flowey/flowey_hvlite/src/pipelines/restore_packages.rs | Changes cfg_versions request from struct to Download enum variant |
| flowey/flowey_hvlite/src/pipelines/mod.rs | Updates clippy lint attribute from cfg_attr to unconditional expect |
| flowey/flowey_hvlite/src/pipelines/custom_vmfirmwareigvm_dll.rs | Changes cfg_versions request from struct to Download enum variant |
| flowey/flowey_hvlite/src/pipelines/checkin_gates.rs | Changes cfg_versions request from struct to Download enum variant |
| flowey/flowey_hvlite/src/pipelines/build_igvm.rs | Adds --use-local-deps and --custom-openvmm-deps CLI parameters; conditionally uses Local or Download cfg_versions request |
| flowey/flowey_hvlite/src/pipelines/build_docs.rs | Changes cfg_versions request from struct to Download enum variant |
| .github/workflows/openvmm-pr.yaml | Updates generated workflow step names from download_openvmm_deps to resolve_openvmm_deps |
| .github/workflows/openvmm-pr-release.yaml | Updates generated workflow step names from download_openvmm_deps to resolve_openvmm_deps |
| .github/workflows/openvmm-ci.yaml | Updates generated workflow step names from download_openvmm_deps to resolve_openvmm_deps |
Comments suppressed due to low confidence (2)
flowey/flowey_lib_hvlite/src/resolve_openvmm_deps.rs:71
- The error message refers to "Path requests" but the actual request variant is named
LocalPath. Consider updating the error message for consistency: "Cannot specify both Version and LocalPath requests".
flowey/flowey_lib_hvlite/src/resolve_openvmm_deps.rs:75 - The error message refers to "Path request" but the actual request variant is named
LocalPath. Consider updating the error message for consistency: "Must specify a Version or LocalPath request".
justus-camp-microsoft
force-pushed
the
local_vmm_deps
branch
from
88bce71 to
ab5ffa1
Compare
justus-camp-microsoft
force-pushed
the
local_vmm_deps
branch
2 times, most recently
from
3f8b4a8 to
2c67f00
Compare
| flowey_request! { | ||
| pub struct Request {} | ||
| pub enum Request { | ||
| Download, |
There was a problem hiding this comment.
This doesn't feel right to me. This feels like something we'd want to set globally, instead of per-pipeline somehow. Like what if I want to run vmm_tests locally? We shouldn't have to add a new command line argument to every single flow, right?
There was a problem hiding this comment.
I agree that it's arduous to add this everywhere, but what's the alternative? If we based it on an environment variable, we still have to check if it's set everywhere. This seems more in line with "flowey best practices" to me but I'm open to other ways of doing this.
For the vmm_tests question, it's not implemented here but we could add something similar to what I added in build_igvm to use all local dependencies. I'm focused on reproducible builds at the moment, but I don't think it would be too much work to add. Another benefit of doing this with an explicit request type is that it makes adding it pretty straightforward, whereas with the global environment variable approach you have to implicitly know where to go check for it.
There was a problem hiding this comment.
Maybe we could enlighten flowey_cli with the concept of global arguments? flowey_hvlite could define a type that's clap-compatible, flowey_cli could become generic over that type, and then the value could get passed into every pipeline in OpenvmmPipelines?
There was a problem hiding this comment.
Looking into doing it that way now - will report back
There was a problem hiding this comment.
So pipelines do have the concept of global variables already, but they're used for job execution related operations rather than any runtime configuration (see --viz-mode). We have a couple "global" variables that get passed in to set the working directory and persisted flowey directory, but that involves setting them in the runtime VarDB and reading them later, which isn't all that different from just checking an environment variable at runtime imo. There also could be edge cases/other nodes that download things in individual pipelines that I might not be aware of, so I think having to implement this on a per-pipeline basis may actually be better for now?
The end-goal is for all flows to go through the "local" dependency path using nix to download the dependencies, so I think long-term this request type and parameter would end up going away as it's eventually going to be the default.
There was a problem hiding this comment.
I don't think we want to modify the current set of global variables, since those are applicable to every possible pipeline. I'm thinking more adding something that would be flowey_hvlite-specific, where we could add something like this, then update every pipeline to require checking them.
Maybe we should wait on this PR until after the planned meeting to go over reproducible builds in full.
justus-camp-microsoft
force-pushed
the
local_vmm_deps
branch
from
2c67f00 to
ac020bd
Compare
justus-camp-microsoft
force-pushed
the
local_vmm_deps
branch
from
ac020bd to
c6f14fd
Compare
justus-camp-microsoft
force-pushed
the
local_vmm_deps
branch
from
c6f14fd to
1a3d0b1
Compare
2 participants
In order to get to reproducible builds, we're going to need all nodes that currently download something at build time to allow for Nix to download them instead and then specify the path at which they've been downloaded. There are several nodes that don't currently have Local override request variants. I've started with
download_openvmm_deps(renamed toresolve_openvmm_deps, but have also plumbed a--use-local-depscommand line parameter to build-igvm for testing locally.