Prevent job termination on Slurm node lookup failures

[faganihajizada]

Summary

This PR fixes a critical bug where IsNodeDrain() and IsNodeDrained() functions use fail-open error handling, causing immediate pod deletion and job termination when SLURM node lookups fail.

Problem:

When the operator cannot query a Slurm node (e.g., node name mismatch, REST API unavailable, or node not yet registered), the current implementation returns true (indicating the node is drained), which causes the operator to immediately delete pods and kill running jobs..

Querying a non-existent Slurm node returns HTTP 200:

{ "errors": [ { "description": "Failure to query node bad-node-name", "error_number": 0, "error": "No error", "source": "_dump_nodes" } ], "warnings": [], "meta": { "plugin": { "type": "openapi\/slurmctld", "name": "Slurm OpenAPI slurmctld", "data_parser": "data_parser\/v0.0.43", "accounting_storage": "" }, "client": { "source": "localhost:6820(fd:11)", "user": "root", "group": "root" }, "command": [], "slurm": { "version": { "major": "25", "micro": "3", "minor": "05" }, "release": "25.05.3", "cluster": "slurm" } } } HTTP_STATUS:200

Solution:

Changed to fail-closed behavior - return false and the error when node status cannot be reliably determined. This prevents premature pod deletion and allows the operator to retry until it can properly verify the node is drained.

This fix makes the failure mode safe: if we cannot verify drain status, we wait and retry rather than destroying potentially busy nodes.

Breaking Changes

N/A

Testing

Deployed operator to verify fail-closed behavior:

{"level":"error","ts":"2025-11-14T15:41:06+01:00",
 "msg":"encountered an error while reconciling request",
 "controller":"nodeset-controller",
 "controllerGroup":"slinky.slurm.net",
 "controllerKind":"NodeSet",
 "NodeSet":{"name":"slurm-worker-slinky","namespace":"slurm"},
 "error":"Not Found",
 "errorCauses":[{"error":"Not Found"}]}