(3.6.0 - latest) Prolog hangs due to long GPU health check times on certain instance types

[hehe7318]

The Issue

AWS ParallelCluster 3.6.0 and later, when configured with GPU health checks, may experience delays and eventual "prolog hung" errors when using instance types like p4d and p5. These instances have a complex GPU topology, which results in lengthy diagnostic checks during the GPU health check process. The Prolog, which runs the check before job tasks are started, must complete on all allocated nodes. If one node's GPU health check takes too long, the entire job setup is delayed, causing the "prolog hung" error:

slurmstepd: error: Prolog hung on node xxx

Testing with g4dn and g6 instance types shows that this issue occurs rarely, as these instances have simpler GPU configurations that do not require such long diagnostic times.

Affected Versions

All ParallelCluster 3.6.0+ versions using Slurm scheduler on instance types such as p4d and p5 with GPU health checks enabled are affected. This issue may also occur with other instance types if the GPU health check takes too long.

Mitigation

You can find a detailed explanation and the mitigation of the problem. (3.6.0 ‐ latest) Prolog hangs due to long GPU health check times on certain instance types

[gwolski]

@hehe7318 In your mitigation explanation, you suggest adding a config option to slurm.conf:

PrologFlags=Alloc,NoHold

Even for non-gpu clusters the prolog takes about 40 seconds. Is there any reason one should not just make this the default? I spin up hundreds of machines per day, and anything I can do to get the user's job running faster is a win.

[hehe7318]

Hi @gwolski, for non-GPU instances, you can disable the GPU health check.
And yes, PrologFlags=Alloc,NoHold can make the prolog run earlier, during resource allocation rather than right before task launch, and the actual computing job will not start until the prolog script complete even with the Nohold Flags.

NoHold
If set, the Alloc flag should also be set. This will allow for salloc to not block until the prolog is finished on each node. The blocking will happen when steps reach the slurmd and before any execution has happened in the step. This is a much faster way to work and if using srun to launch your tasks you should use this flag. This flag cannot be combined with the Contain or X11 flags.

So in your scenario, it truly can help you to get the user's job running faster and I cannot see a disadvantage.

Is there any reason one should not just make this the default?

I can consider this as a feature request. We have never evaluated this approach. It will need a security check.