Separate VMs step on each other's port assignments

[thomasjm]

Description

I noticed that when I start two separate Lima VMs and try to start Docker containers on them with automatically assigned host ports, they end up listening to the same ports (!).

It seems to happen silently. To repro, first start a container called docker1:

limactl start template://docker --name docker1
export DOCKER_HOST=$(limactl list docker1 --format 'unix://{{.Dir}}/sock/docker.sock')
docker run -d -p 80 nginx
docker ps -a

...
CONTAINER ID   IMAGE     COMMAND                  CREATED        STATUS        PORTS                                       NAMES
224e25506de7   nginx     "/docker-entrypoint.…"   1 second ago   Up 1 second   0.0.0.0:32768->80/tcp, [::]:32768->80/tcp   silly_tu

Now, let's do it a second time with the name docker2:

limactl start template://docker --name docker2
export DOCKER_HOST=$(limactl list docker2 --format 'unix://{{.Dir}}/sock/docker.sock')
docker run -d -p 80 nginx
docker ps -a
...
CONTAINER ID   IMAGE     COMMAND                  CREATED                  STATUS                  PORTS                                       NAMES
a82de0b1db16   nginx     "/docker-entrypoint.…"   Less than a second ago   Up Less than a second   0.0.0.0:32768->80/tcp, [::]:32768->80/tcp   goofy_euler

As you can see, now I have two containers that both claim to be forwarding port 32768. In practice I believe the first one wins, and you aren't able to access the second container. Also, it's odd that it is picking exactly 2^15 for the number.

I'm on Lima 1.1.1 and macOS Sequoia 15.5 (Apple silicon).

[afbjorklund]

This is a known shortcoming, and also happens if you run two Kubernetes VMs for instance (both listening on 6443)

You can manually assign different ports to the containers, but currently the VM doesn't "know" about the host ports.

i.e. host services will also "hide" port running in a VM, like if you deploy a web server on 8080 and have one on the host

[afbjorklund]

it's odd that it is picking exactly 2^15 for the number.

EDIT: I think the number comes from the /proc/sys/net/ipv4/ip_local_port_range, so maybe configure that differently?

The docker daemon keeps an internal state of which ports have been allocated, so it needs to be restarted as well...

[jandubois]

Maybe we can automate this. We can keep an index of each running VM in their instance directory, e.g.

$ cat $LIMA_HOME/default/index
2

So when we start a new VM we check all running instances, and then pick the lowest available index for the new VM, but no larger than 28.

In a boot script we modify ip_local_port_range to (32+index)*1000 60999.

That way each instance has 1000 ports for itself before it starts overlapping with the next instance.

[jandubois]

In a boot script we modify ip_local_port_range to (32+index)*1000 60999.

I don't know if code may pick a port randomly from the pool, in which case we should probably also set the high value to the low value+1000.

And while I think 28 running instances is plenty, we could also give the first 10 instances 1000 ports, the next 20 get 500 ports, and then the rest gets increments of 100.

That way you get plenty of ports if you have only a few instances, but fewer, if you have a lot. Which seems a reasonable compromise. But maybe this is over-engineering things. 😄

[thomasjm]

host services will also "hide" port running in a VM, like if you deploy a web server on 8080 and have one on the host

If I understand correctly, this only works if a host service binds a more general interface like 0.0.0.0, while the Lima port forwarding binds a less general interface. This seems tricky and I wouldn't want to rely on it...

And while I think 28 running instances is plenty, we could also give the first 10 instances 1000 ports, the next 20 get 500 ports, and then the rest gets increments of 100.

Likewise, this kind of scheme isn't generally correct, because you could still easily collide with some other port being used on the host.

I'd suggest that generally the only correct way to get a random host port for a port forwarding agent is to ask the OS for a free port. Although in this case I'm not sure how you'd do it. I understand that Lima works by watching for iptables events in the guest, and then setting up ssh -L tunnels. (I don't know how the gRPC tunnels work so I'll just talk about the SSH version.)

My first thought was that you could do ssh -L 0:localhost:80 and SSH would automatically get an OS port, but it doesn't seem to support that.

One possibility could be a SOCKS proxy? You do ssh -D 8080 <target>, and then you can query arbitrary ports on the remote host like this: curl --socks5 localhost:8080 http://internal-service.com. Annoyingly, this doesn't support passing 0 for an auto-allocated port either. But once you have the SOCKS proxy, you can set up a userspace port-forwarding process using a SOCKS-aware proxying tool like Caddy (or just write your own) that will claim a random host port.

Of course, what this gives up is that the port on the host matches the port on the VM. Users would need a way to query Lima for the mapping in order to use something like this.

[AkihiroSuda]

SOCKS

We already have limactl tunnel for this
https://lima-vm.io/docs/reference/limactl_tunnel/

[jandubois]

this kind of scheme isn't generally correct, because you could still easily collide with some other port being used on the host.

It is not; it is just a heuristic to make collisions less likely. Currently they are guaranteed, as you found out.

the only correct way to get a random host port for a port forwarding agent is to ask the OS for a free port

Yes, but that requires co-operation from the app opening the port. So it is not possible in general if you have no control over the program opening the port. However, see below...

Of course, what this gives up is that the port on the host matches the port on the VM.

Yes, but we are already talking about a randomly allocated port here.

Users would need a way to query Lima for the mapping in order to use something like this.

Yes, I think that would be good anyways, if e.g. limactl ls --json would include a list of currently mapped ports.

Then we would need an extension to portForwards in lima.yaml where we can set guestPort: any, and the host agent would just allocate a random available port.

Right now you would have to look at ha.stderr.log to determine the mapping, but we should make the mapping available in structured form via limactl ls for automation.

[afbjorklund]

Having two Docker VMs (or two Kubernetes VMs) sounds like a niche use case, not sure it needs a generic solution?

Having different ports on the host and in the guest breaks the "illusion", and shows the machine behind the curtain.

[thomasjm]

Yes, but that requires co-operation from the app opening the port. So it is not possible in general if you have no control over the program opening the port.

No not at all. Forgive my ignorance if I'm misunderstanding something, but I'm picturing it like this:

1. The process on the VM opens a port like normal.
2. Lima detects this by monitoring the VM.
3. Lima sets up port forwarding, by asking the OS for a host port and setting up an appropriate proxy.

No need to change the program opening the port. This would be a collision-free and race condition-free way to do things.

Yes, but we are already talking about a randomly allocated port here.

Right, but mirroring the ports is a nice property to have because it simplifies the picture. For example, when using Docker, you can create a container with a randomly chosen port, run docker ps etc. to find out what it is, and then that's also your host port. Unfortunately, I don't think it's a workable picture in general (see below).

Having two Docker VMs (or two Kubernetes VMs) sounds like a niche use case, not sure it needs a generic solution?

Strongly disagree. For my use-case, I want to make an app that can be deployed against a Docker VM. End users will only need to run a single VM. But I still want the ability to test multiple instances of them in parallel for CI etc. Same thing with Kubernetes VMs, if I am using Lima to create a development or testing box to develop an app against Kubernetes, then I'm probably going to want to create multiple in parallel in order to test them concurrently.

Imagine if the authors of Docker had said "creating multiple containers on the same port sounds like a niche use case, and besides, breaking the 1:1 port mapping between host and container breaks the illusion, so we don't need to do port mapping." It would severely limit what you can do with Docker.

Also, even if we grant that multiple instances is niche: I'd argue that the way it works right now is simply not correct if the host may have even 1 other service running. When VM processes are free to open whatever port they want, it is not possible to guarantee that the same port number will be available on the host. So something like Docker may choose a random port in the VM that collides with a port in use on the host. While this is unlikely (but not impossible) in simple dev scenarios, it presents big problems if I want to use Lima on a busy CI server, for example.

[afbjorklund]

You can still run multiple instances, just wondered how many things need to be built-in and how much can be added-on.

[afbjorklund]

If you run your VMs with a network setting that gives them a unique IP, and just not NAT, that could be one alternative (to use the full address of the VM, and not just a port). There are also other complimentary options, such as NoRouter.io

[jandubois]

Yes, but that requires co-operation from the app opening the port. So it is not possible in general if you have no control over the program opening the port.

No not at all. Forgive my ignorance if I'm misunderstanding something, but I'm picturing it like this:

1. The process on the VM opens a port like normal.
2. Lima detects this by monitoring the VM.
3. Lima sets up port forwarding, by asking the OS for a host port and setting up an appropriate proxy.

No need to change the program opening the port. This would be a collision-free and race condition-free way to do things.

I was talking about allocating a port inside the VM that is free on the host to have the same port number in both places. Lima tries to maintain the illusion that there is no VM, but everything is available directly on the host.

[thomasjm]

If you run your VMs with a network setting that gives them a unique IP, and just not NAT, that could be one alternative

I tried this and it works pretty well, so I think this is the solution I'll go with.

I found another problem with the port forwarding, which is that I couldn't reliably detect when a port forward was active and ready to go. I have code that starts up a Lima VM and a Docker container, and then immediately tries to connect to it from the host. This fails intermittently, even when I added a wait/polling process to wait until the host TCP port is available. The only reliable way I found to do this was to add a fixed delay of a few seconds. I think it might go through some state where the host TCP port is allocated, but it's not yet connected to the other side.

Lima tries to maintain the illusion that there is no VM, but everything is available directly on the host.

I think you can already tell I'm not a fan of this approach :).

Docker has a mode where you pass --network=host, and then the container directly uses the host network interface. I believe it's the least commonly used networking mode in Docker, since Docker has more powerful options like port forwarding and user-defined networks etc. Lima's port forwarding at the moment is like running with --network=host, but without the safety of actually being operating on the same network interface, so it is vulnerable to all kinds of issues like shadowing, race conditions, problems starting up and shutting down the VM. A casual glance through the issue tracker shows issues like this: #2122, #2536.

I understand other tools like Rancher Desktop have dynamic port forwarding too, so maybe it's a feature users expect and can't be done away with. But I would humbly suggest a couple ways this situation could be improved:

1. Surface errors somehow in Lima when port bindings are not behaving as expected, i.e. the host port was already in use when Lima tried to bind it. I think having the VM fail to start up at all would be better than allowing mysterious port shadowing to occur.
2. Perhaps shift the emphasis away from port forwarding and towards vzNAT-based networking with the VM's IP address. Making it easier to use by displaying the IP address in limactl ls output would be a great start (a la Report VM IP addresses to host agent #3616).