Add details about non-migratable VMs

  How VM is marked as non-migratable
  How VM is processed on upgrade process
  How VM is processed on node maintenance process
  How `migrate` menu comes

Signed-off-by: Jian Wang <[email protected]>

Author: w13915984028

docs/host/host.md

@@ -23,6 +23,8 @@ Because Harvester is built on top of Kubernetes and uses etcd as its database, t
 
 Admin users can enable Maintenance Mode (select **⋮ > Enable Maintenance Mode**) to automatically evict all virtual machines from a node. This mode leverages the **live migration** feature to migrate the virtual machines to other nodes, which is useful when you need to reboot, upgrade firmware, or replace hardware components. At least two active nodes are required to use this feature.
 
+Check if there are any [non-migratable VMs](../vm/live-migration.md#non-migratable-vms) and take essential actions.
+
 :::warning
 
 A [bug](https://github.com/harvester/harvester/issues/7128) may cause an I/O error to occur in virtual machines while Maintenance Mode is enabled on the underlying node. To mitigate the issue, you can set a taint on the node before enabling Maintenance Mode.
@@ -135,19 +137,7 @@ Eviction cannot be completed if the remaining nodes cannot accept replicas from
 
 ### 4. Manage non-migratable VMs.
 
-[Live migration](../vm/live-migration.md) cannot be performed for VMs with certain properties.
-
-- The VM has PCI passthrough devices or vGPU devices.
-
-  A PCI device is bound to a node. You must remove the PCI device from the VM, or delete the VM and then create a new VM from a backup or snapshot.
-
-- The VM has a node selector or affinity rules that bind it to the node to be removed.
-
-  You must change the node selector or affinity rules.
-
-- The VM is on a VM network that binds it to the node to be removed.
-
-  You must select a different VM network.
+Check if there are any [non-migratable VMs](../vm/live-migration.md#non-migratable-vms) and take essential actions.
 
 :::tip
 Create a backup or snapshot for each non-migratable VM before modifying the settings that bind it to the node that you want to remove.

docs/upgrade/automatic.md

@@ -77,6 +77,22 @@ The Harvester and Rancher upgrade processes are independent of each other. Durin
 
 When a Rancher version reaches its End of Maintenance (EOM) date, Harvester only provides fixes for critical security-related issues that affect integration functions (Virtualization Management). For more information, see the [Harvester & Rancher Support Matrix](https://www.suse.com/suse-harvester/support-matrix/all-supported-versions/).
 
+## VM management through the upgrade
+
+### Live-migratable VMs
+
+Those VMs are migrated to other nodes automatically when the hosting node is to be upgraded, they have zero down-time through the upgrade.
+
+### Non-migratable VMs
+
+When an upgrade is triggered, Harvester does a couple of checks, and depends on the value of [upgrade-config setting option `restoreVM`](../advanced/settings.md#upgrade-config):
+
+- False: Harvester refuses the upgrade when any [non-migratable VM](../vm/live-migration.md#non-migratable-vms) is still running. You need to power off them manually.
+
+- True: Harvester will power off those [non-migratable VM](../vm/live-migration.md#non-migratable-vms) when the node is upgraded and then restore them after the node is rebooted.
+
+See [Phase 4: Upgrade Nodes](./troubleshooting.md#phase-4-upgrade-nodes) for more details.
+
 ## Before starting an upgrade
 
 Check out the available [`upgrade-config` setting](../advanced/settings.md#upgrade-config) to tweak the upgrade strategies and behaviors that best suit your cluster environment.

docs/upgrade/troubleshooting.md

@@ -93,7 +93,7 @@ If the upgrade fails at this point, you must generate a [support bundle](../trou
 The Harvester controller creates the following jobs on each node:
 
 - Multi-node clusters:
-  - `pre-drain` job: Live-migrates or shuts down virtual machines on the node. Once completed, the embedded Rancher service upgrades the RKE2 runtime on the node.
+  - `pre-drain` job: [Live-migrates or shuts down virtual machines](./automatic.md#vm-management-through-the-upgrade) on the node. Once completed, the embedded Rancher service upgrades the RKE2 runtime on the node.
   - `post-drain` job: Upgrades and reboots the operating system.
 - Single-node clusters:
   - `single-node-upgrade` job: Upgrades the operating system and RKE2 runtime. The job name uses the format `hvst-upgrade-xxx-single-node-upgrade-<hostname>`.
@@ -277,4 +277,4 @@ New images are loaded to each Harvester node during upgrades. When disk usage ex
 
 If you encounter the error message `Node xxx will reach xx.xx% storage space after loading new images. It's higher than kubelet image garbage collection threshold 85%.`, run `crictl rmi --prune` to clean up unused images before starting a new upgrade.
 
-![Disk space not enough error message](/img/v1.4/upgrade/disk-space-not-enough-error-message.png)
+![Disk space not enough error message](/img/v1.4/upgrade/disk-space-not-enough-error-message.png)

docs/vm/create-vm.md

@@ -209,8 +209,48 @@ It is also possible to connect VMs using additional networks with Harvester's bu
 In bridge VLAN, virtual machines are connected to the host network through a linux `bridge`. The network IPv4 address is delegated to the virtual machine via DHCPv4. The virtual machine should be configured to use DHCP to acquire IPv4 addresses.
 
 ## Node Scheduling
+
 `Node Scheduling` allows you to constrain which nodes your VMs can be scheduled on based on node labels.
 
+![vm-node-scheduling](/img/v1.6/vm/vm-node-scheduling.png)
+
+There are three options:
+
+- Run virtual machine on any aviailable node
+
+- Run virtual machine on specific node
+
+  Following example shows the VM targets a node with `hostname harv21`:
+
+  ```
+        nodeSelector:
+          kubernetes.io/hostname: harv21
+  ```
+
+- Run virtual machine on node(s) matching scheduling rules
+
+  A flexiable option to customize the VM to be scheduled to a group of nodes. Following example shows the VM targets those nodes with label key `harvesterhci.io/group` and value `engineering` or `qa`.
+
+  ```
+      spec:
+        affinity:
+          nodeAffinity:
+            requiredDuringSchedulingIgnoredDuringExecution:
+              nodeSelectorTerms:
+                - matchExpressions:
+                    - key: harvesterhci.io/group
+                      operator: In
+                      values:
+                        - engineering
+                        - qa
+  ```
+
+:::note
+
+The VM might be [non-migratable](./live-migration.md#non-migratable-vms) when `Run virtual machine on specific node` is selected.
+
+:::
+
 See the [Kubernetes Node Affinity Documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity) for more details.
 
 ## VM Scheduling
@@ -296,6 +336,12 @@ No affinity rules are applied when a virtual machine connects to VM networks tha
 
 :::
 
+:::note
+
+The VM might be [non-migratable](./live-migration.md#non-migratable-vms) when there is only one node participates in the `cluster network`.
+
+:::
+
 #### Related CPU Pinning Concepts
 
 When you enable the [CPU Manager](./cpu-pinning.md#enable-and-disable-cpu-manager) on nodes, Harvester applies the following label to related `node` objects.
@@ -325,6 +371,12 @@ spec:
                       - 'true'
 ```
 
+:::note
+
+The VM might be [non-migratable](./live-migration.md#non-migratable-vms) when the `CPU Manager` is only enabled on one node.
+
+:::
+
 ## Annotations
 
 Harvester allows you to attach custom metadata to virtual machines using annotations. These key-value pairs enable extended features or behaviors without requiring changes to the core virtual machine configuration.

docs/vm/live-migration.md

@@ -17,15 +17,37 @@ description: Live migration means moving a virtual machine to a different host w
 
 Live migration means moving a virtual machine to a different host without downtime.
 
-:::note
+## Non-migratable VMs
 
-- Live migration is not allowed when the virtual machine is using a management network of bridge interface type.
-- Live migration is not allowed when the virtual machine has any volume of the `CD-ROM` type. Such volumes should be ejected before live migration.
-- Live migration is not allowed when the virtual machine has any volume of the `Container Disk` type. Such volumes should be removed before live migration.
-- Live migration is not allowed when the virtual machine has any `PCIDevice` passthrough enabled. Such devices need to be removed before live migration.
-- Live migration is not allowed when the volumeAccessMode of any volume in the virtual machine is `ReadWriteOnce`. Such volumes should be removed before live migration.
+The definitions of VM are versatile, a VM cannot perform live migration when one or more of following conditions are met.
 
-:::
+Remove the related device or add more schedulable nodes can make the VM live-migratable.
+
+### Has non-migratable devices or node-selector
+
+- The VM has any volume of the `CD-ROM` type.
+
+- The VM has any volume of the `Container Disk` type.
+
+- The VM has any volume with `volumeAccessMode` `ReadWriteOnce`.
+
+- The VM has `PCI passthrough` or `vGPU` devices.
+
+- The VM has a [node selector](./create-vm.md#node-scheduling) that binds it to a specific node.
+
+### Has scheduling rules which can only match one node
+
+Following conditions are checked on the runtime (e.g. before an upgrade) to mark the VM is non-migratable if only one node matches.
+
+- The VM is on a `cluster network` which spreads to only one node.
+
+  See [Automatically Applied Affinity Rules](./create-vm.md#related-networking-concepts) for more details.
+
+- The VM has `cpu-pinning` enabled and there is only one node enables `CPU Manager`.
+
+  See [Automatically Applied Affinity Rules](./create-vm.md#related-cpu-pinning-concepts) for more details.
+
+- Other [node scheduling](./create-vm.md#node-scheduling) rules.
 
 ## How Migration Works
 
@@ -51,6 +73,18 @@ However, `host-model` only allows migration of the VM to a node with same CPU mo
 
 ![](/img/v1.2/vm/migrate-action.png)
 
+:::note
+
+The `Migrate` menu is not available when:
+
+- This is a single-node cluster.
+
+- The VM is `non-migratable` due to it [has non-migratable devices or node-selector](#has-non-migratable-devices-or-node-selector).
+
+- The VM already has a running or pending migration process.
+
+:::
+
 When you have [node scheduling rules](./create-windows-vm.md#node-scheduling-tab) configured for a VM, you must ensure that the target nodes you are migrating to meet the VM's runtime requirements. The list of nodes you get to search and select from will be generated based on:
 - VM scheduling rules.
 - Possibly node rules from the network configuration.
@@ -62,6 +96,14 @@ When you have [node scheduling rules](./create-windows-vm.md#node-scheduling-tab
 1. Go to the **Virtual Machines** page.
 1. Find the virtual machine in migrating status that you want to abort. Select **⋮ > Abort Migration**.
 
+:::note
+
+- The `Abort Migration` menu is available when the VM already has a running or pending migration process.
+
+- Don't click `Abort Migration` if it is triggered by the [Harvester upgrade](../upgrade/automatic.md#live-migratable-vms) or [node maintenance](../host/host.md#node-maintenance).
+
+:::
+
 ## Migration Timeouts
 
 ### Completion Timeout
@@ -88,4 +130,4 @@ Migrating a VM with `host-model` is not possible because the values of `host-mod
 - Cluster level: Run `kubectl edit kubevirts.kubevirt.io -n harvester-system` and add `spec.configuration.cpuModel: "123"`. This change also affects newly created VMs.
 - Individual VMs: Modify the VM configuration to include `spec.template.spec.domain.cpu.model: "123"`.
 
-Both methods require the restarting the VMs. If you are certain that all nodes in the cluster support a specific CPU model, you can define this at the cluster level before creating any VMs. In doing so, you eliminate the need to restart the VMs (to assign the CPU model) during live migration.
+Both methods require the restarting the VMs. If you are certain that all nodes in the cluster support a specific CPU model, you can define this at the cluster level before creating any VMs. In doing so, you eliminate the need to restart the VMs (to assign the CPU model) during live migration.