Introduce Leader Election and use Deployments for agents

[davidepasquero]

The agents have been changed from pod to deployment so that their life cycle is managed by Kubernetes

[mergify]

This pull request is now in conflict. Could you fix it @davidepasquero? 🙏

[davidepasquero]

Context

This pull request implements the approach discussed in the previous PR #54 (comment) and aims to provide a robust, Kubernetes-native solution to the agent lifecycle management problem.

Problem Solved

The previous agent management via single Pods had a critical flaw: in the event of a node crash, the agent pod would get stuck in a "Terminating" state on that node. This prevented it from being rescheduled onto a healthy node without manual intervention, causing a service disruption.

Implemented Solution

Following the feedback received, this PR introduces two fundamental changes:

1. Transition from Pod to Deployment:

   • The agents are no longer managed as individual Pods but through a Kubernetes Deployment.
   • This delegates the responsibility of managing the pod lifecycle to the Kubernetes Deployment controller. If a pod (or its node) fails, the kubernetes deploy controller will automatically create a new replica on an available node, ensuring high availability.

2. Introduction of Leader Election:

   • Since a Deployment can manage multiple replicas, a leader election mechanism has been introduced to ensure that only one agent instance (the "leader") is active at any given time.
   • The other instances remain on standby, ready to take over if the current leader fails.
   • This prevents conflicts and ensures that operations performed by the agent are not duplicated.

Benefits of this Approach

• Reliability: Definitively solves the problem of stuck pods.
• Kubernetes-Native Standard: Aligns the agent's behavior with the best practices and patterns commonly adopted in the Kubernetes and Harvester ecosystem.
• Simplified Management: It leverages native Kubernetes mechanisms, reducing the need for custom logic to handle failures.

[davidepasquero]

Key Technical Changes

1. Fixed Agent Leader Election Namespace

• Problem: Initially, the agent attempted to create its leader election lock (a ConfigMap or Lease) in the kube-system namespace, causing permission errors.
• Solution:
  • Modified the agent's code (cmd/agent/run.go) to use its own namespace for leader election. This namespace is obtained via the POD_NAMESPACE environment variable, injected through the Downward API.
  • Updated the agent's deployment manifest (chart/templates/agent-deployment.yaml) to provide this environment variable to the pod.

2. Fixed Webhook Certificate SAN Mismatch

• Problem: Resolved a TLS error where the webhook certificate was not valid for the expected FQDN (...webhook.harvester-system.svc vs. ...webhook..svc). The issue was that the harvester/webhook library was not receiving the correct namespace during certificate generation.
• Solution: Added the --namespace {{ .Release.Namespace }} argument to the webhook container in its deployment manifest (chart/templates/deployment.yaml) to provide the correct namespace.

3. Dynamic Agent Network Interface Management (Multus NAD Attachment)

• Feature: Implemented functionality for the main vm-dhcp-controller to dynamically update the vm-dhcp-agent Deployment when an IPPool is detected.
• Implementation:
  • Controller RBAC: Extended the main controller's RBAC permissions to allow it to get, list, watch, patch, and update Deployment resources in its namespace.
  • IPPool Controller Logic: Modified pkg/controller/ippool/controller.go to:
    • Fetch the agent's Deployment.
    • Add/update the k8s.v1.cni.cncf.io/networks annotation to specify the NAD (derived from IPPool.spec.networkName) to which the agent must connect.
    • Update the --nic container argument with the Multus interface name (e.g., net1).
    • Update the IPPOOL_REF environment variable in the agent container with the namespace/name of the current IPPool.
  • Agent Static IP Configuration on Multus Interface:
    • Added --server-ip and --cidr CLI flags to the agent, which are set by the main controller based on the IPPool's configuration.
    • Added the NET_ADMIN capability to the agent container to allow it to configure network interfaces.
    • Implemented logic in pkg/agent/agent.go (configureInterface method) so that the agent, upon becoming leader, configures the specified network interface with the provided static IP and CIDR.

4. Agent DHCP Lease Cache Synchronization on Startup/Failover

• Problem: Fixed an issue where a newly elected leader agent might not find existing leases because its DHCP server started before the internal lease cache was synchronized with the IPPool's state.
• Solution:
  • Introduced a synchronization mechanism (InitialSyncDone channel and sync.Once) to control the startup sequence.
  • The agent's local controller now first populates the DHCPAllocator cache by reading IPPool.Status.IPv4.Allocated and then signals that the initial sync is complete.
  • The agent (pkg/agent/agent.go) now waits for this signal before starting the actual DHCP server.
  • Added an AddFunc to the agent's informer to ensure the IPPool's initial state is processed immediately on startup to populate the cache.

5. Miscellaneous Build Fixes

• Resolved several Go build errors related to missing/duplicate imports, unused variables, and incorrect logic (e.g., LeaseTime handling, references to removed constants).

[starbops]

Hi @davidepasquero. Could you help organize your commits and provide a concise summary of what this PR is trying to achieve and how it works? Thank you.

[davidepasquero]

Hi, Certainly, please allow me a little time. I will begin working on this next week. Thank you very much. Davide Il giorno mer 9 lug 2025 alle ore 08:03 Zespre Chang < ***@***.***> ha scritto:

…

***@***.**** commented on this pull request. Hi @davidepasquero <https://github.com/davidepasquero>. Could you help organize your commits and provide a concise summary of what this PR is trying to achieve and how it works? Thank you. — Reply to this email directly, view it on GitHub <#61 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANZRW3D67S7ELFO2QSHE7ST3HSWDDAVCNFSM6AAAAACAYCIQYSVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZTAMBQGA4DMNJYGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[davidepasquero]

Hi @davidepasquero. Could you help organize your commits and provide a concise summary of what this PR is trying to achieve and how it works? Thank you.

#64