Lima Driver Registry and Discovery

[jandubois]

Lima Driver Registry and Discovery

These are my thoughts how I expect Lima drivers to work once we have both built-in and external drivers:

There is no extra driver registry configuration file

limactl will discover installed drivers automatically. External drivers have names like lima-driver-virtualbox. Only the first driver of a specific name is registered. The priority sequence is:

• compiled-in drivers
• drivers located in the same directory as limactl
• drivers located via the PATH directory

Driver implementations don't have hard-coded names

Drivers have a default name under which they register, but they are only used when compiled into limactl. External driver names are derived from the executable name. If you rename the file, the driver name changes as well.

built-in drivers are registered via an init function

If the driver source is included in the compilation, it automatically registers its NewDriver function with its built-in name. If you include an additional driver in the compilation of limactl then it will automatically be built-in.

external drivers are registered with an RPC wrapper

For external drivers we will register a NewDriver function that is a closure of the name and path, and implements an RPC shim to the external driver.

Drivers don't need to know if they are built-in or external

The source code of a driver does not need to be changed to build it internally or externally; it is just controlled by build options. We may use a --tags drivers tag to include all in-tree drivers. Compiling without the tag will result in a limactl that has no built-in drivers.

We will have a single "driver host"

Compiling cmd/lima-driver together with any actual driver implementation will produce an executable for that driver.

There is basically a single driver that has self-registered with its default name (inside the driver host). The driver host will update the registration with the driver name derived from the executable name, and then expose the driver via RPC.

The user can get a list of available drivers

• limactl create --list-drivers shows a list off all driver names.
• limactl info will include a drivers property with both names of all drivers, and paths for external drivers.

[AkihiroSuda]

both built-in and external drivers:

Why do you need built-in drivers?
For saving the total binary footprint?

drivers located via the PATH directory

Could be also <PREFIX>/libexec/lima/plugins/vm/<SHORTNAME>.

• Implement VM plugin subsystem #3384 (comment)

No strong opinion though

limactl create --list-drivers shows a list off all driver names.

• This can be slightly expensive if the plugins are put in $PATH
• limactl info should just suffice?

[jandubois]

Why do you need built-in drivers? For saving the total binary footprint?

We don't need them. I thought I remembered that either you or @afbjorklund said before that default drivers should still be built-in.

I kind of prefer single-executable tools, but that ship has already sailed by having separate guest agent binaries.

drivers located via the PATH directory

Could be also <PREFIX>/libexec/lima/plugins/vm/<SHORTNAME>.

Yeah, that is maybe better. Except it will be inconvenient if you develop a driver outside the Lima repo and want to test it without installing it. We could add LIMA_DRIVERS_PATH for this purpose.

limactl create --list-drivers shows a list off all driver names.

• This can be slightly expensive if the plugins are put in $PATH

Why would it be slow? It should take no longer than it takes the shell (or the OS) to run a command located via the PATH (or doing tab-completion on lima-driver-).

Anyways, --list-drivers is just a companion to --list-templates. In some ways it is redundant to tab completion, but not everyone installs tab completion in their shell.

$ limactl start --list-drivers
qemu
vz
virtualbox

Ideally we would also use it for tab completion on --vm-type.

• limactl info should just suffice?

Given that limactl info only produces JSON I think it is more targeted to automation, and to Lima maintainers, not regular users.

[afbjorklund]

The built-in vs external model was mostly built on docker-machine, which served as my mental model for these new drivers.

Also my goal was to change as little possible for the existing drivers, while making it possible to also add new ones...

[jandubois]

I think there is a bit of a naming disconnect between lima-driver and --vm-type; I would prefer them to use similar names.

I think I would prefer renaming --vm-type to --driver and vmOpts to driverOpts (and allowing the old names for backwards compatibility).

I prefer driver to plugin, which has been suggested elsewhere, but I guess that is a personal preference based on similarity to docker-machine-driver etc. I think lima-driver-qemu is better than lima-plugin-qemu. And plugin would also have the problem that we would need to rename --vm-type and vmOpts.

lima-vm-type-qemu and lima-vm-qemu are not appealing to me, but would be possible choices if we don't want to change --vm-type and vmOpts.

Other ideas?

[jandubois]

Possible, but what other drivers to we anticipate to have?

I'm aiming for 2 things:

1. Consistency
2. Short names that accurately correspond to the concept

So for consistency reasons I don't want to have Driver and BaseDriver types in the code, that represent plugins at the executable level and are selected by a --vm-type. That is just confusing for everyone, and especially people new to the project.

Short names are generally better because they are faster to type, and can be combined with other names to create property names that are still not overly long.

For that reason I prefer driver and plugin over interface and extension.

And while I have a slight personal preference for driver, I will be fine with plugin if we are going to use the name consistently and not also have drivers, and vm types to refer to exactly the same thing.

We will have to continue to support --vm-type and vmOpts for backwards compatibility reasons anyways, so I fully understand the desire not to change them, but it is still battling with my need for consistency. 😄

[unsuman]

Possible, but what other drivers to we anticipate to have?

Drivers like VirtualBox, vfkit etc?

Consistency is important, for sure! We can go for plugin(since Lima is closely related to containerd).
Here is my thoughts of what needs to be changed if we opt for plugin, correct me if I am wrong:

• A new VMPlugin interface will be created that will embed the interface driver.Driver with plugin related functions. But do we need to create another interface or should we just use the current driver.Driver? If we don't need another interface we can rename driver.Driver --> driver.VMPlugin.
• Struct driver.BaseDriver --> driver.BaseVMPlugin or simply driver.BasePlugin.
• Standalone binaries will go into <PREFIX>/libexec/lima/plugins/vm/<SHORTNAME> and LIMA_PLUGIN_PATH for plugins outside of lima repo.
• --vm-type --> --vm-plugin + backward compatibility.
• vmOpts --> pluginVMOpts + backward compatibility.

Let me know your thoughts!

[AkihiroSuda]

--vm-type and vmOpts can be just used for drivers/plugins without renaming.

[jandubois]

Possible, but what other drivers to we anticipate to have?

Drivers like VirtualBox, vfkit etc?

Sorry, I meant other kinds of drivers. These would all be instances of the same driver type.

Consistency is important, for sure! We can go for plugin(since Lima is closely related to containerd). Here is my thoughts of what needs to be changed if we opt for plugin, correct me if I am wrong:

I would contest that "Lima is closely related to containerd". It is has some builtin support for installing containerd, but otherwise is a generic Linux Machine provisioning tool.

But let's say we settle on plugin:

• A new VMPlugin interface will be created that will embed the interface driver.Driver with plugin related functions. But do we need to create another interface or should we just use the current driver.Driver? If we don't need another interface we can rename driver.Driver --> driver.VMPlugin.

I don't think the interface should be called VMPlugin but just Plugin. The plugin mechanism should work for bare metal machines as well, as long as they support cloud-init, and a way for us to start them. These are "Linux Machine Plugins", so we should just call them "Plugin".

Also that's why I asked what other types of drivers/plugins should we anticipate. I think it is unlikely that there are any, but let's say we have network plugins¹. Then we could call that interface "NetworkPlugin" and still use the bare "Plugin" name for Lima plugins.

• Struct driver.BaseDriver --> driver.BaseVMPlugin or simply driver.BasePlugin.

I wonder if the BaseDriver would not simply become the Plugin type, and Driver would be an implementation detail of the plugin that isn't exposed to Lima directly at all.

• Standalone binaries will go into <PREFIX>/libexec/lima/plugins/vm/<SHORTNAME> and LIMA_PLUGIN_PATH for plugins outside of lima repo.

I would drop the /vm subdirectory; I don't think it is needed.

And I would prefer to use a longer name, like lima-plugin-qemu instead of just qemu. Avoids confusion and makes debugging easier, even if it is just things like ps -ef | grep lima-plugin.

I think either the variable should be LIMA_PLUGINS_PATH, or we should rename LIMA_TEMPLATES_PATH to LIMA_TEMPLATE_PATH. Since you called it <PREFIX>/libexec/lima/plugins (and we have /usr/local/share/lima/templates) I think the plural form should be used for the environment variables to match the file system path names.

• --vm-type --> --vm-plugin + backward compatibility.
• vmOpts --> pluginVMOpts + backward compatibility.

Let's not do this for now, and see how the rest turns out first.

Footnotes

1. In any case. I would expect network plugins to be plugins specific for a driver, and not for Lima. ↩

[afbjorklund]

I have no strong opinion on "plugins" vs "drivers", but I don't think we have to rename --vm-type to --driver

I don't like the idea of renaming the drivers to "plugins", and you could always add --driver as a flag alias.

[unsuman]

The three drivers: Qemu, vz and wsl2 are currently present in the github.com/lima-vm/lima/pkg repo. These drivers should be moved out of the main repo like github.com/lima-vm/qemu, github.com/lima-vm/vz, github.com/lima-vm/wsl2. If that's the case how should I proceed this in a PR?

[AkihiroSuda]

I prefer monorepo for these official drivers

[unsuman]

I prefer monorepo for these official drivers

Monorepo as in a single repository that contains all three drivers together, rather than having separate repositories for each driver.

[AkihiroSuda]

Yes

[jandubois]

I would prefer to keep the existing drivers in the main repo, and just build separate binaries for them, just like we handle the guestagent.

Moving them to a separate repo just makes development harder for no benefit. I like the simplicity of being able to do git clone; cd lima; make native install and having a working installation.
And even if you don't want to install the new build, you can test it out of _output.

Especially during development of the driver/plugin mechanism when changes will have to be done both to Lima and to the drivers this will make the edit-debug cycle more complex than it needs to be.

We can always move the default drivers out of the repo once the new mechanism has settled.

[afbjorklund]

These drivers should be moved out of the main repo

I don't think moving the drivers was part of the feature, and would be happy to clear up any misunderstanding that lead to it.

The only thing we said was that it would be possible to add new drivers, without having to make a change in the upstream repository first. Some projects, like minikube, have a hardcoded list of "supported" drivers which make that impossible... I prefer the model in docker-machine, where you didn't need "permission" to add a new driver. And those new drivers exist in their own git repositories, and only use a "SDK" from the upstream project in order to compile their driver binaries to the right version.

I also didn't understand why they would need separate binaries?

[jandubois]

Since we had a Zoom call, and also some discussion on Slack, I want to summarize what we (@AkihiroSuda, @afbjorklund, @unsuman, and @jandubois) have agreed on so far:

• We will use the term "driver"
• We will continue to us --vm-type and vmOpts
• Current drivers remain in the lima.git repo; new drivers will each have their own repo
• Drivers can be builtin or separate executables based on build config
• Drivers can be located via "builtin", by LIMA_PLUGINS_PATH, same path as limactl, by PATH, or in .../libexec.

[unsuman]

Also,

• Removing driver.BaseDriver struct.