feat: nupm registry subcommand

[mkatychev]

This PR introduces the registry subcommand for nupm to allow registry management and support multiple registries:

Usage:
  > nupm registry

Subcommands:

  nupm registry add - Add a new registry
  nupm registry describe - Show detailed information about a specific registry
returning a list of package names, type, and version
  nupm registry fetch - Fetch and cache registry data locally
  nupm registry init - Initialize a new nupm registry or a registry index if the `--index` flag is
passed in
  nupm registry list - List all configured registries
  nupm registry remove - Remove a registry
  nupm registry rename - Rename a registry
  nupm registry set-url - Update a given registry url

[kubouch]

Seems like you're adding a new concept of "registry index" file that replaces the $env.NUPM_REGISTRIES. I think that's fine, some thoughts:

• We should probably remove $env.NUPM_REGISTRIES completely from the codebase to avoid having multiple competing concepts.
• It would be good to explain in https://github.com/nushell/nupm/blob/main/docs/design/registry.md what the index is. The nupm registry command help message could also have a small explainer how it works.
• Storing it in $env.NUPM_HOME is fine. I would, however, manage it automatically, ie., not requiring the separate "init" step. I think it should be created together with NUPM_HOME and populated with the default content.
• Store the registry_idx.nuon name in a constant at the top of the file in case we want to rename it in the future.
• What is the difference between registry add and registry new that you have in the TODO?

[mkatychev]

@kubouch thanks for the feedback, do note this is a draft PR so

• What is the difference between registry add and registry new that you have in the TODO

I could use some feedback here but I see three usecases when dealing with registries that aren't covered yet:

• initialize a new registry (registry new? perhaps registry init is better)
• add a registry to the registry index (registry add)
• initialize a package (package init?)

[kubouch]

I'd say for consistency, if we initialize something, let's call it init. Hence:

• Initialize a new registry: nupm registry init
• Add a registry to the index: nupm registry add
• Initialize a new package: nupm init (by default nupm operates on packages so no need to add a package IMO)

[Rydwxz]

Seems like you're adding a new concept of "registry index" file that replaces the $env.NUPM_REGISTRIES. I think that's fine, some thoughts:

I think we should prefer environment variables over filesystem items for configuration of nupm, for these reason:

• there should be one consistent easy to use method to configure all the different parts of nupm
• that way should be as close to the same way nushell itself is configured as reasonably possible, not hidden in nupm internals
• the contents of $NUPM_HOME should be managed by nupm, not the user, and we should not assume that it will always be inside $XDG_CONFIG_HOME
• allows it to be set by parent processes for security or testing reasons without complicated secondary hierarchical loading system

Since we may collect a few more variables, a nested record like this could simply be added to the end of or sourced by one of the existing config files:

$env.nupm_config = { 
    nupm_home: "/home/rw/.config/nushell/nupm" # can still be a default
    registries: { 
        nupm: "github.com/nushell/nupm/registry.nuon" 
        awesome_nu: "..." 
    }
    sync_list: { ... }
    sync_on_launch: false
}

[mkatychev]

@Rydwxz are you suggesting registries to be added manually? I think doing programmatic addition/removal of registries could reduce some UX friction as well as allow for some sanitisation of what goes into the record.

[Rydwxz]

I want to be able to have a fully declarative configuration that uses config.nu as an entry point. But I don't want to force $NUPM_HOME to be in the same place. To that end, $NUPM_HOME/registry_list.nuon can be a source of truth about what registries have been added interactively/imperatively, but it can't be the only source of truth: nupm registry list should append the list it finds in registry_list.nuon to the list it finds in $env.NUPM_REGISTRIES, does that make sense?

[mkatychev]

@Rydwxz @kubouch this is ready for review. Since there's already a lot to review I would like to defer commands to initialize registries and packages into a separate PR:

nupm/nupm/registry.nu

Lines 241 to 251 in baaf4f4

	# Initialize a new nupm registry or a registry index if the `--index` flag is
	# passed in
	@example "Initialize registry index" { nupm registry init --index }
	@example "Initialize registry list" { nupm registry init-index }
	export def init [--index] {
	if $index {
	init-index
	return
	}
	# TODO initialize registry index here
	}

[Rydwxz]

Hi @mkatychev, first of all thank you for making all those changes to accomodate what I asked! The declarative configuration that I was worried about can be achieved without it, so I am worried that I accidentally added too much to the scope of this PR when I mentioned changing to a nested environment variable. I think it's the most sensible/scalable way to organize things, but it might be a breaking change for people who are manually setting those variables right now.

[mkatychev]

@Rydwxz I can revert the nested config structure in this PR.
I think that the nested config structure is a good idea, should it be introduced in a follow-up PR?
The experimental disclaimer at the top of this repo's README should make it acceptable to introduce this breaking change.

[mkatychev]

@Rydwxz I've reverted the requested changes, this should be ready for review once more

[kubouch]

General typos also to a separate file, please, otherwise the commit history becomes a mess.

[mkatychev]

Apologies, thought this was a squash merge repo, I will reword

[kubouch]

Sorry, I meant, could you move the .md files into a separate PRs? These changes are unrelated to adding the nupm registry subcommand. Yes, we squash, which would result in these typos being labeled as "nupm registry subcommand" -- this can be annoying eg. when using Git blame.

[kubouch]

Sorry for the late reply, I'm not available as much as I used to be. I think some of these would be better moved to a separate PR. Also left some misc comments.

I would also leave out fetch, see the comment. I also don't see it used in the PR, let's not add things speculatively (unless I missed something).

About the registries, I think having only the index file should be fine. With the $env.NUPM_REGISTRIES we run into the problem of serializing it between nu invocations, and I don't think we really need it. I added it initially as a just to have something running quickly, but it wasn't meant to be a long-term solution.

A big part of future nupm is virtual environments using overlays, similar to what I sketched in https://github.com/kubouch/nuun. Overlay would define NUPM_HOME and all the stuff derived from it, such as PATH, the registry index file, etc. Here is an example overlay-creation code (very bare-bones). Some things are better left global, like cache -- the point of cache is to reuse stuff between virtual environments. But I think each virtual environment should have its own registry index, and handling them via overlays lets you set it from config.nu, or wherever. Just activate the desired overlay.

I hope it's a bit clearer now.

Otherwise, seems good, the registry subcommands can be expanded/tweaked as necessary in future PRs.

[mkatychev]

I would also leave out fetch, see the comment. I also don't see it used in the PR, let's not add things speculatively (unless I missed something).

Agreed, my mistake on not removing it, realized it was there to largely test private registries.

A big part of future nupm is virtual environments using overlays, similar to what I sketched in https://github.com/kubouch/nuun. Overlay would define NUPM_HOME and all the stuff derived from it, such as PATH, the registry index file, etc.

Would probably be good to get an overlay test at some point.

[mkatychev]

@kubouch hopefully this should address most of the issues.

[kubouch]

I think you're missing the registry subdirectory here. It now dumps the files into $env.NUPM_CACHE/<registry-name> instead of $env.NUPM_CACHE/registry/<registry-name>.

[kubouch]

Here too, I think, missing registry subdirectory.

[kubouch]

Here it should print something like "nupm registry init without --index is not implemented yet".

[kubouch]

Here (and other commands with the --save flag), the print should reflect what happened depending on the --save flag, for example:

• $"Registry '($name)' added successfully. To make the change permanent by saving it to the registry index, use the --save flag."
• $"Registry '($name)' added successfully and saved to ($env.NUPM_INDEX_PATH)."

[kubouch]

OK, I think it's good, except a few minor comments. We can merge it with the NUPM_REGISTRIES and refactor the system as we implement the overlays.

I originally thought that it would be better to defer implementing overlays until the base features work reliably well, but given how central overlays are to the overall design, I'm thinking that should be the next step to implement.

[kubouch]

If you could fix the last few comments, we can merge it then.