Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

296 v1alpha1 #598

Closed
wants to merge 69 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
d782656
Make Ident a generic type
rydrman Oct 9, 2022
5197d99
Update recipe and package to have different idents
rydrman Oct 10, 2022
ddc22f9
Clean up ident impls and relations
rydrman Oct 13, 2022
db5bceb
Remove debug statements
rydrman Nov 19, 2022
e89392d
Add skelton of v1 spec type
rydrman Nov 5, 2022
3a67d0e
Add code to deserialize and serialize the example spec
rydrman Nov 12, 2022
d4b7b20
Break large option file into multiple
rydrman Nov 14, 2022
7c0b01b
Expand example test to entire recipe spec
rydrman Nov 14, 2022
2a5f845
Add missing top-level fields to the recipe spec
rydrman Nov 14, 2022
44505d4
Fix bug parsing envop with separator field
rydrman Nov 15, 2022
5336b52
Allow components to use undeclared default ones
rydrman Nov 15, 2022
5dcdda5
Use assert_eq that shows better data diffs
rydrman Nov 15, 2022
b952904
Fill out structs to parse the remaining example spec
rydrman Nov 15, 2022
4b4db3f
Move v0 structures to v0 crate
rydrman Nov 15, 2022
de766e8
Move test definition behind a trait
rydrman Nov 15, 2022
f296c9d
Formalize embedded packages as a generic type
rydrman Nov 16, 2022
171f8be
Remove use of v0::Opt from binary builder
rydrman Nov 18, 2022
05fd021
Add conditional logic for when blocks
rydrman Nov 18, 2022
6b2584b
Fail for unknown fields in testing
rydrman Nov 18, 2022
272707a
Allow components to be conditional in v1 recipes
rydrman Nov 18, 2022
c700bde
Expand BuildEnv trait to include options
rydrman Nov 19, 2022
f3be021
Add a Variant trait that's not just an option map
rydrman Nov 22, 2022
ee61693
Add new api variants to FromYaml implementation
rydrman Dec 5, 2022
3f4719f
Add source pkg generation for v1
rydrman Dec 5, 2022
fc7b67c
Add function to resolve uses and get full component
rydrman Dec 7, 2022
db90767
Make runtime_reqts function take a set of components
rydrman Dec 7, 2022
574f8f8
Add build enviroment calculation to v1 recipes
rydrman Dec 11, 2022
a4c7379
Add resolved options to v1 package type
rydrman Dec 11, 2022
7696b58
Add method to auto-handle namespaces in optionmap get
rydrman Dec 15, 2022
bceafee
Implement build option resolving for v1 recipe
rydrman Dec 15, 2022
f2de044
Do not generate empty var requests for v1 specs
rydrman Dec 15, 2022
ff75976
Update build script to be a Cow String
rydrman Dec 15, 2022
5ff3987
Add logic to generate v1 binary package
rydrman Dec 15, 2022
b2346ab
Add logic to get downstream requirements from v1 packages
rydrman Dec 15, 2022
b595d87
Add validation spec to v1 package
rydrman Dec 15, 2022
e2e760c
Update script block to properly parse a simple string or conditional
rydrman Dec 15, 2022
cd226d2
Ensure default components in v1 spec
rydrman Dec 15, 2022
eed82db
Update requirements.contains to check all items before giving up
rydrman Dec 15, 2022
d3dd75f
Add error when build script is found empty
rydrman Dec 15, 2022
32a2111
Add default variant to v1 recipe
rydrman Dec 15, 2022
9b5776c
Remove error implementation for PkgRequest
rydrman Dec 15, 2022
c72180c
Implement var request satisfy for v1 package
rydrman Dec 15, 2022
d44d619
Improve solution data structure/ownership
rydrman Dec 16, 2022
51eac62
Add default value for package script
rydrman Dec 16, 2022
65056fe
Flatten v1 package struct
rydrman Dec 16, 2022
b688a12
Remove 'atDownstreamBuild from pkg options
rydrman Dec 16, 2022
95e8643
Expand build env to report used components
rydrman Dec 16, 2022
734844c
Add static var assignement support to NameAndValue
rydrman Dec 18, 2022
b6c29c0
Remove atDownstreamBuild altogether
rydrman Dec 18, 2022
f2f3c18
Ressurect auto injection of downstream requirements for v0
rydrman Dec 18, 2022
fb3cc0a
Implement additional requirements for v1 variants
rydrman Dec 18, 2022
8204c58
Inject downstream requirements into v1 components
rydrman Dec 19, 2022
ea09a3f
Add option validation for v1 packages
rydrman Dec 20, 2022
3530f37
Remove unused impl Satisfy for recipe types
rydrman Dec 20, 2022
8d7d154
Add placeholders to catch common errors in yaml
rydrman Dec 20, 2022
3a181aa
Add lint command to validate yaml files
rydrman Dec 20, 2022
8907e76
Fix spec files that won't lint
rydrman Dec 20, 2022
e6a4c00
Add check to disable publishing unstable apis by default
rydrman Dec 20, 2022
23c5a40
Actually assign a version number for downstream requests in v1
rydrman Dec 21, 2022
31d6439
Fix pkg options not propagating downstream
rydrman Dec 21, 2022
0d4dc01
Use exact version matching for @source
rydrman Dec 22, 2022
0a0b565
Actually apply options as runtime requirements in packages
rydrman Dec 22, 2022
f8c5e13
Update BuildEnv to have a target
rydrman Dec 22, 2022
f1ffefc
Allow for options to be conditional on current package components
rydrman Dec 22, 2022
6e4c33c
Update v1 options to also resolve values for pkg options
rydrman Dec 23, 2022
5b67610
Update bootstrap packages to include version number, be v1 specs
rydrman Dec 23, 2022
8b09a9f
Convert some packages to v1 spec as examples/testing
rydrman Dec 23, 2022
750ed00
Fix bug where source package was always regenerated in makefile
rydrman Dec 23, 2022
fd36197
Make docker easier to control from interactive shell
rydrman Dec 23, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Make Ident a generic type
The pattern is <base>/<target> which allows for defining different
spcificity levels as well as nice composition of different ident types.
This initial commit defines different ident types as well as an AnyIdent
with an optional build that replaces the old ident nearly everywhere.
Next steps are to use more specific identifier types where applicable

Signed-off-by: Ryan Bottriell <[email protected]>
  • Loading branch information
rydrman committed Nov 15, 2022
commit d782656fdd389a461a182c3dcec989001ebfdb89
37 changes: 20 additions & 17 deletions crates/spk-build/src/build/binary.rs
Original file line number Diff line number Diff line change
@@ -17,8 +17,8 @@ use spk_schema::foundation::ident_component::Component;
use spk_schema::foundation::name::OptNameBuf;
use spk_schema::foundation::option_map::OptionMap;
use spk_schema::foundation::version::VERSION_SEP;
use spk_schema::ident::{PkgRequest, PreReleasePolicy, RangeIdent, RequestedBy};
use spk_schema::{ComponentFileMatchMode, ComponentSpecList, Ident, Package, PackageMut};
use spk_schema::ident::{PkgRequest, PreReleasePolicy, RangeIdent, RequestedBy, VersionIdent};
use spk_schema::{AnyIdent, ComponentFileMatchMode, ComponentSpecList, Package, PackageMut};
use spk_solve::graph::Graph;
use spk_solve::solution::Solution;
use spk_solve::{BoxedResolverCallback, DefaultResolver, ResolverCallback, Solver};
@@ -96,7 +96,8 @@ where
{
/// Create a new builder that builds a binary package from the given recipe
pub fn from_recipe(recipe: Recipe) -> Self {
let source = BuildSource::SourcePackage(recipe.to_ident().into_build(Build::Source).into());
let source =
BuildSource::SourcePackage(recipe.to_ident().with_target(Some(Build::Source)).into());
Self {
recipe,
source,
@@ -368,11 +369,9 @@ where
) -> Result<HashMap<Component, spfs::encoding::Digest>> {
self.build_artifacts(package, options).await?;

let source_ident = Ident {
name: self.recipe.name().to_owned(),
version: self.recipe.version().clone(),
build: Some(Build::Source),
};
let source_ident =
VersionIdent::new(self.recipe.name().to_owned(), self.recipe.version().clone())
.into_any(Some(Build::Source));
let sources_dir = data_path(&source_ident);

let mut runtime = spfs::active_runtime().await?;
@@ -555,8 +554,7 @@ where
env.insert(
"SPK_PKG_BUILD".to_string(),
spec.ident()
.build
.as_ref()
.build()
.map(Build::to_string)
.unwrap_or_default(),
);
@@ -614,7 +612,7 @@ where
}

fn split_manifest_by_component(
pkg: &Ident,
pkg: &AnyIdent,
manifest: &spfs::tracking::Manifest,
components: &ComponentSpecList,
) -> Result<HashMap<Component, spfs::tracking::Manifest>> {
@@ -652,7 +650,12 @@ fn split_manifest_by_component(
}
for node in manifest.walk() {
if relevant_paths.contains(&node.path) {
tracing::debug!("{}:{} collecting {:?}", pkg.name, component.name, node.path);
tracing::debug!(
"{}:{} collecting {:?}",
pkg.name(),
component.name,
node.path
);
let mut entry = node.entry.clone();
if entry.is_dir() {
// we will be building back up any directory with
@@ -670,39 +673,39 @@ fn split_manifest_by_component(
}

/// Return the file path for the given source package's files.
pub fn source_package_path(pkg: &Ident) -> RelativePathBuf {
pub fn source_package_path(pkg: &AnyIdent) -> RelativePathBuf {
data_path(pkg)
}

/// Return the file path for the given build's spec.yaml file.
///
/// This file is created during a build and stores the full
/// package spec of what was built.
pub fn build_spec_path(pkg: &Ident) -> RelativePathBuf {
pub fn build_spec_path(pkg: &AnyIdent) -> RelativePathBuf {
data_path(pkg).join("spec.yaml")
}

/// Return the file path for the given build's options.json file.
///
/// This file is created during a build and stores the set
/// of build options used when creating the package
pub fn build_options_path(pkg: &Ident) -> RelativePathBuf {
pub fn build_options_path(pkg: &AnyIdent) -> RelativePathBuf {
data_path(pkg).join("options.json")
}

/// Return the file path for the given build's build.sh file.
///
/// This file is created during a build and stores the bash
/// script used to build the package contents
pub fn build_script_path(pkg: &Ident) -> RelativePathBuf {
pub fn build_script_path(pkg: &AnyIdent) -> RelativePathBuf {
data_path(pkg).join("build.sh")
}

/// Return the file path for the given build's build.sh file.
///
/// This file is created during a build and stores the bash
/// script used to build the package contents
pub fn component_marker_path(pkg: &Ident, name: &Component) -> RelativePathBuf {
pub fn component_marker_path(pkg: &AnyIdent, name: &Component) -> RelativePathBuf {
data_path(pkg).join(format!("{}.cmpt", name))
}

2 changes: 1 addition & 1 deletion crates/spk-build/src/build/sources_test.rs
Original file line number Diff line number Diff line change
@@ -141,6 +141,6 @@ async fn test_sources_environment(_tmpdir: tempfile::TempDir) {
let actual = std::fs::read_to_string(out_file).unwrap();
assert_eq!(
actual, expected,
"should have access to package variables in sources script"
"should have access to package variables in sources script, want: {expected}, got: {actual}"
);
}
4 changes: 2 additions & 2 deletions crates/spk-cli/cmd-install/src/cmd_install.rs
Original file line number Diff line number Diff line change
@@ -81,7 +81,7 @@ impl Run for Install {
println!(" Requested:");
for spec in primary {
let mut end = String::new();
if spec.ident().build.is_none() {
if spec.ident().build().is_none() {
end = " [build from source]".magenta().to_string();
}
println!(" {}{end}", spec.ident().format_ident());
@@ -91,7 +91,7 @@ impl Run for Install {
}
for spec in tertiary {
let mut end = String::new();
if spec.ident().build.is_none() {
if spec.ident().build().is_none() {
end = " [build from source]".magenta().to_string();
}
println!(" {}{end}", spec.ident().format_ident())
2 changes: 1 addition & 1 deletion crates/spk-cli/cmd-make-binary/src/cmd_make_binary.rs
Original file line number Diff line number Diff line change
@@ -187,7 +187,7 @@ impl Run for MakeBinary {
std::env::current_dir().context("Failed to get current directory")?;
builder.with_source(BuildSource::LocalPath(here));
} else if let Some(PackageSpecifier::WithSourceIdent((_, ref ident))) = package {
// Use the source package `Ident` if the caller supplied one.
// Use the source package `AnyIdent` if the caller supplied one.
builder.with_source(BuildSource::SourcePackage(ident.clone()));
}
let out = match builder.build_and_publish(&local).await {
6 changes: 3 additions & 3 deletions crates/spk-cli/cmd-make-source/src/cmd_make_source.rs
Original file line number Diff line number Diff line change
@@ -11,7 +11,7 @@ use spk_build::SourcePackageBuilder;
use spk_cli_common::{flags, CommandArgs, Run};
use spk_schema::foundation::format::FormatIdent;
use spk_schema::foundation::spec_ops::Named;
use spk_schema::ident::BuildIdent;
use spk_schema::ident::LocatedBuildIdent;
use spk_schema::{Package, Recipe, SpecTemplate, Template, TemplateExt};
use spk_storage::{self as storage};

@@ -41,7 +41,7 @@ impl Run for MakeSource {
}

impl MakeSource {
pub async fn make_source(&mut self) -> Result<Vec<BuildIdent>> {
pub async fn make_source(&mut self) -> Result<Vec<LocatedBuildIdent>> {
let _runtime = self
.runtime
.ensure_active_runtime(&["make-source", "mksource", "mksrc", "mks"])
@@ -89,7 +89,7 @@ impl MakeSource {
idents.push(
out.ident()
.clone()
.try_into_build_ident(local.name().to_owned())?,
.try_into_located_build_ident(local.name().to_owned())?,
);
}
Ok(idents)
2 changes: 1 addition & 1 deletion crates/spk-cli/cmd-test/src/cmd_test.rs
Original file line number Diff line number Diff line change
@@ -189,7 +189,7 @@ impl Run for Test {
BuildSource::SourcePackage(
recipe
.to_ident()
.into_build(Build::Source)
.with_target(Some(Build::Source))
.into(),
)
},
7 changes: 4 additions & 3 deletions crates/spk-cli/cmd-test/src/test/build.rs
Original file line number Diff line number Diff line change
@@ -13,7 +13,7 @@ use spk_schema::foundation::ident_build::Build;
use spk_schema::foundation::ident_component::Component;
use spk_schema::foundation::option_map::OptionMap;
use spk_schema::ident::{PkgRequest, PreReleasePolicy, RangeIdent, Request, RequestedBy};
use spk_schema::{Ident, Recipe, SpecRecipe};
use spk_schema::{AnyIdent, Recipe, SpecRecipe};
use spk_solve::graph::Graph;
use spk_solve::solution::Solution;
use spk_solve::{BoxedResolverCallback, DefaultResolver, ResolverCallback, Solver};
@@ -36,7 +36,8 @@ pub struct PackageBuildTester<'a> {

impl<'a> PackageBuildTester<'a> {
pub fn new(recipe: SpecRecipe, script: String) -> Self {
let source = BuildSource::SourcePackage(recipe.to_ident().into_build(Build::Source).into());
let source =
BuildSource::SourcePackage(recipe.to_ident().with_target(Some(Build::Source)).into());
Self {
prefix: PathBuf::from("/spfs"),
recipe,
@@ -155,7 +156,7 @@ impl<'a> PackageBuildTester<'a> {
self.execute_test_script(&source_dir, env, &rt)
}

async fn resolve_source_package(&mut self, package: &Ident) -> Result<Solution> {
async fn resolve_source_package(&mut self, package: &AnyIdent) -> Result<Solution> {
let mut solver = Solver::default();
solver.update_options(self.options.clone());
let local_repo: Arc<storage::RepositoryHandle> =
4 changes: 2 additions & 2 deletions crates/spk-cli/cmd-test/src/test/sources.rs
Original file line number Diff line number Diff line change
@@ -103,7 +103,7 @@ impl<'a> PackageSourceTester<'a> {
if self.source.is_none() {
// we only require the source package to actually exist
// if a local directory has not been specified for the test
let source_pkg = self.recipe.to_ident().into_build(Build::Source);
let source_pkg = self.recipe.to_ident().with_target(Some(Build::Source));
let mut ident_range = RangeIdent::equals(&source_pkg, [Component::Source]);
ident_range.components.insert(Component::Source);
let request = PkgRequest::new(ident_range, RequestedBy::SourceTest(source_pkg))
@@ -132,7 +132,7 @@ impl<'a> PackageSourceTester<'a> {

let source_dir = match &self.source {
Some(source) => source.clone(),
None => source_package_path(&self.recipe.to_ident().into_build(Build::Source))
None => source_package_path(&self.recipe.to_ident().with_target(Some(Build::Source)))
.to_path(&self.prefix),
};

8 changes: 4 additions & 4 deletions crates/spk-cli/common/src/flags.rs
Original file line number Diff line number Diff line change
@@ -16,7 +16,7 @@ use spk_schema::foundation::name::{OptName, OptNameBuf};
use spk_schema::foundation::option_map::{host_options, OptionMap};
use spk_schema::foundation::spec_ops::Named;
use spk_schema::foundation::version::CompatRule;
use spk_schema::ident::{parse_ident, Ident, PkgRequest, Request, RequestedBy, VarRequest};
use spk_schema::ident::{parse_ident, AnyIdent, PkgRequest, Request, RequestedBy, VarRequest};
use spk_schema::{Recipe, SpecTemplate, Template, TemplateExt, TestStage};
use spk_solve::{self as solve};
use spk_storage::{self as storage};
@@ -228,7 +228,7 @@ impl Requests {
&self,
options: &OptionMap,
packages: I,
) -> Result<Vec<Ident>> {
) -> Result<Vec<AnyIdent>> {
let mut idents = Vec::new();
for package in packages {
if package.contains('@') {
@@ -237,7 +237,7 @@ impl Requests {

match stage {
TestStage::Sources => {
let ident = recipe.to_ident().into_build(Build::Source);
let ident = recipe.to_ident().with_target(Some(Build::Source));
idents.push(ident);
continue;
}
@@ -307,7 +307,7 @@ impl Requests {

match stage {
TestStage::Sources => {
let ident = recipe.to_ident().into_build(Build::Source);
let ident = recipe.to_ident().with_target(Some(Build::Source));
out.push(PkgRequest::from_ident(ident, RequestedBy::CommandLine).into());
}

6 changes: 3 additions & 3 deletions crates/spk-cli/common/src/publish.rs
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ use std::sync::Arc;

use spk_schema::foundation::format::{FormatComponents, FormatIdent};
use spk_schema::foundation::ident_component::ComponentSet;
use spk_schema::{Ident, Package, Recipe};
use spk_schema::{AnyIdent, Package, Recipe};
use spk_storage::{self as storage};
use storage::{with_cache_policy, CachePolicy};

@@ -71,7 +71,7 @@ impl Publisher {
}

/// Publish the identified package as configured.
pub async fn publish(&self, pkg: &Ident) -> Result<Vec<Ident>> {
pub async fn publish(&self, pkg: &AnyIdent) -> Result<Vec<AnyIdent>> {
let recipe_ident = pkg.with_build(None);
tracing::info!("loading recipe: {}", recipe_ident.format_ident());
match with_cache_policy!(self.from, CachePolicy::BypassCache, {
@@ -118,7 +118,7 @@ impl Publisher {
}
}

let builds = if pkg.build.is_none() {
let builds = if pkg.build().is_none() {
with_cache_policy!(self.from, CachePolicy::BypassCache, {
self.from.list_package_builds(pkg)
})
6 changes: 3 additions & 3 deletions crates/spk-cli/group1/src/cmd_deprecate.rs
Original file line number Diff line number Diff line change
@@ -9,7 +9,7 @@ use clap::Args;
use colored::Colorize;
use spk_cli_common::{flags, CommandArgs, Run};
use spk_schema::foundation::format::FormatIdent;
use spk_schema::ident::{parse_ident, Ident};
use spk_schema::ident::{parse_ident, AnyIdent};
use spk_schema::{Deprecate, DeprecateMut, Package, Recipe, Spec, SpecRecipe};
use spk_storage::{self as storage};

@@ -167,7 +167,7 @@ pub(crate) async fn change_deprecation_state(

let ident = parse_ident(name)?;
for (repo_name, repo) in repos.iter() {
if ident.build.is_none() {
if ident.build().is_none() {
match repo.read_recipe(&ident).await {
Ok(recipe) => {
to_action.push((
@@ -355,7 +355,7 @@ impl DeprecateMut for DeprecationTarget {
}

impl DeprecationTarget {
fn ident(&self) -> Ident {
fn ident(&self) -> AnyIdent {
match self {
DeprecationTarget::Recipe(r) => r.to_ident(),
DeprecationTarget::Package(r) => r.ident().clone(),
12 changes: 6 additions & 6 deletions crates/spk-cli/group2/src/cmd_ls.rs
Original file line number Diff line number Diff line change
@@ -13,7 +13,7 @@ use spk_cli_common::{flags, CommandArgs, Run};
use spk_schema::foundation::format::{FormatComponents, FormatIdent, FormatOptionMap};
use spk_schema::foundation::ident_component::ComponentSet;
use spk_schema::foundation::name::{PkgName, PkgNameBuf};
use spk_schema::ident::{parse_ident, Ident};
use spk_schema::ident::{parse_ident, AnyIdent};
use spk_schema::ident_ops::parsing::{ident_parts, IdentParts, KNOWN_REPOSITORY_NAMES};
use spk_schema::{Deprecate, Package, Spec};
use spk_storage::{self as storage};
@@ -277,8 +277,8 @@ impl<T: Output> Ls<T> {
for (package, index) in packages {
let (repo_name, repo) = repos.get(index).unwrap();
let mut versions = {
let base = Ident::from(package);
repo.list_package_versions(&base.name)
let base = AnyIdent::from(package);
repo.list_package_versions(base.name())
.await?
.iter()
.filter_map(|v| match search_term {
@@ -301,7 +301,7 @@ impl<T: Output> Ls<T> {
..
}) = search_term
{
if let Some(this_build) = &build.build {
if let Some(this_build) = build.build() {
if search_build != this_build.to_string() {
continue;
}
@@ -340,7 +340,7 @@ impl<T: Output> Ls<T> {

async fn format_build(
&self,
pkg: &Ident,
pkg: &AnyIdent,
spec: &Spec,
repo: &storage::RepositoryHandle,
) -> Result<String> {
@@ -351,7 +351,7 @@ impl<T: Output> Ls<T> {

// Packages without builds, or /src packages have no further
// info to display
if pkg.build.is_none() || pkg.is_source() {
if pkg.build().is_none() || pkg.is_source() {
return Ok(item);
}

4 changes: 2 additions & 2 deletions crates/spk-cli/group2/src/cmd_publish.rs
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ use std::sync::Arc;
use anyhow::Result;
use clap::Args;
use spk_cli_common::{CommandArgs, Publisher, Run};
use spk_schema::Ident;
use spk_schema::AnyIdent;
use spk_storage::{self as storage};

/// Publish a package into a shared repository
@@ -38,7 +38,7 @@ pub struct Publish {
/// This can be an entire package version with all builds or a
/// single, specific build.
#[clap(name = "PKG", required = true)]
pub packages: Vec<Ident>,
pub packages: Vec<AnyIdent>,
}

#[async_trait::async_trait]
Loading