Cozystack doesn't recover from system reboot.

[marekrogosz]

Describe the bug
I'm trying to setup a 3 bare metal node cluster. I started with one node and plan to add two more once the first one is ready.

However I noticed rebooting the machine (either gracefully or via power cycle) results in Cozystack not booting again. I need my homelab to recover from power interruptions.

Environment

• Cozystack version: 0.38.4
• Provider: on-prem

To Reproduce
Steps to reproduce the behavior:

1. Deploy Cozystack on a single node as per documentation.
2. Observe all tenant-root pods are in Ready state.
3. Reboot the machine (either gracefully or via power cycle).
4. Wait for machine to boot up again.

Expected behavour
Cozystack should come back online and cluster should be available.

Actual behaviour
Talos comes back online but Cozystack dashboard is not available.

Logs
I noticed etcd pods are missing after reboot:

❯ kubectl get pod -n tenant-root
NAME                                          READY   STATUS      RESTARTS   AGE
alerta-c98d86f94-sb8zx                        0/1     Completed   0          57m
alerta-c98d86f94-vwgbs                        0/1     Completed   0          66m
alerta-db-1                                   0/1     Completed   0          57m
alerta-db-2                                   0/1     Completed   0          56m
cm-acme-http-solver-lzb9k                     0/1     Error       0          66m
cm-acme-http-solver-zdvkt                     0/1     Error       0          66m
grafana-db-1                                  0/1     Completed   0          57m
grafana-db-2                                  0/1     Completed   0          56m
grafana-deployment-768b84ffcd-48kvr           0/1     Completed   5          66m
grafana-deployment-768b84ffcd-6j2sk           0/1     Completed   3          57m
grafana-deployment-768b84ffcd-6l64l           0/1     Completed   4          66m
grafana-deployment-768b84ffcd-jgrvx           0/1     Completed   4          57m
root-ingress-controller-75c59d8c84-stsvx      0/2     Completed   2          69m
root-ingress-controller-75c59d8c84-vjzgp      0/2     Completed   2          69m
root-ingress-defaultbackend-cd98c755b-56pfl   0/1     Completed   0          69m
root-ingress-defaultbackend-cd98c755b-kfppz   0/1     Completed   0          57m
vlogs-generic-5f54c7f9d4-ngmlf                0/1     Completed   0          66m
vmalert-vmalert-shortterm-5c58dd9f5b-fmhnq    0/2     Completed   0          66m
vmalert-vmalert-shortterm-5c58dd9f5b-g54tg    0/2     Completed   0          57m
vminsert-longterm-6b4565b447-5cpgv            0/1     Completed   1          63m
vminsert-longterm-6b4565b447-8b4s8            0/1     Completed   0          57m
vminsert-longterm-6b4565b447-cxxrn            0/1     Completed   0          62m
vminsert-shortterm-5fc4d4b977-6p5kq           0/1     Completed   0          63m
vminsert-shortterm-5fc4d4b977-ptkgp           0/1     Completed   0          62m
vminsert-shortterm-5fc4d4b977-rxpsh           0/1     Completed   0          57m
vminsert-shortterm-5fc4d4b977-x282x           0/1     Completed   0          57m
vmstorage-longterm-1                          0/1     Error       1          66m

Screenshots
If applicable, add screenshots to help explain the problem.

Additional context
I'm new to Cozystack and Kubernetes so forgive me my lack of knowledge.

I also tried to follow Talos Disaster Recovery steps but that didn't help.

Some more questions:

1. Is the issue caused by my (temporary) single node setup?
2. Do I need to follow Cozystack Backup Restore steps after each reboot?

Checklist

• I have checked the documentation
• I have searched for similar issues
• I have included all required information
• I have provided clear steps to reproduce
• I have included relevant logs

[dosubot]

Hi @marekrogosz! I'm Dosu and I’m helping the cozystack team.

This behavior is expected with a single-node Cozystack setup. The official documentation and issues confirm that production deployments require at least three nodes for high availability and automatic recovery—single-node, non-HA installations are not formally supported and may not recover after a reboot, especially if etcd pods fail to start or their persistent volumes are lost or corrupted [source].

Missing etcd pods after reboot means the control plane cannot function, which is why the dashboard is unavailable. Backup and restore using Velero is intended for disaster recovery, not for routine reboots in a healthy cluster. You do not need to perform backup/restore after every reboot, but in single-node setups, if etcd data is lost, manual recovery from backup is your only option [source].

For a homelab or on-prem environment, the best practice is to add at least two more nodes as soon as possible to achieve a supported, resilient cluster. This will allow etcd to run in HA mode and recover from node failures or reboots as expected [source]. If you must run single-node for now, ensure your storage is reliable and take regular etcd and Velero backups, but be aware that recovery is not guaranteed in this configuration.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[marekrogosz]

As I said I plan to install on 3 nodes but given it's homelab environment all three will be connected to the same power supply.

Will reboots be tolerated better on a 3-node setup when all 3 nodes go down and up at the same time?

I am not too concerned about HA as it's for personal projects but I need to ensure I can recover from a power outage and I don't lose my cluster.

[dosubot]

A properly configured 3-node Cozystack cluster should recover from a full power outage or simultaneous reboot, as long as the persistent storage for etcd and other critical components remains intact and uncorrupted. This is the standard expectation for Kubernetes clusters with replicated etcd and storage: when all nodes come back up, etcd regains quorum and the control plane resumes operation, so you shouldn't need to perform manual backup/restore steps after every routine reboot or outage in a healthy setup docs.

However, homelab environments with all nodes on a single power supply and consumer-grade hardware do carry extra risk. If the disks storing etcd data or LINSTOR replicas are damaged or corrupted during a power loss, recovery may require restoring from backup. To minimize risk, regularly verify that your persistent volumes are healthy, test your recovery procedures, and keep up-to-date Velero and etcd backups for disaster recovery storage backup.

No special steps are required for routine reboots in a healthy 3-node cluster, but robust storage and regular backups are your safety net in case of hardware failure or data corruption.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[lllamnyp]

Hi @marekrogosz , thanks for reporting. This looks a lot like an issue with kube-ovn. I'll see if I can reproduce this.