Accelerate SLURM job state queries with --only-job-state option to squeue

[tcutts]

New feature

Make SLURM requests more efficient by [optionally] using the --only-job-state option to squeue

Use case

SLURM supported releases (24 and 25) introduced the ability to return just the job state information much more quickly. Nextflow should use this capability if it's available in order to reduce the RPC burden on SLURM itself and accelerate scheduling.

Suggested implementation

All that needs doing is changing the squeue command line in the executor plugin to:

squeue --noheader --only-job-state -o "%i %t" -t all

(i.e. one extra option). The format of the output is almost identical; I think the only difference is whether the % syntax for job arrays is emitted (i.e. how many jobs in the job array can be run at once, something I don't think nextflow makes use of)

If the SLURM administrator combines this with adding the enable_job_state_cache configuration in SLURM, it will mean that nextflow job state queries will no longer use the RPC mechanism which causes such performance difficulties for naive workflows.

Obviously there will need to be a bit more code than that; either to make the precise command configurable or to test the SLURM version for whether that option is available, and use it if it is.

[bentsherman]

PRs are welcome! The challenge we always run into is that we don't always have a SLURM cluster available to test these changes, so it's easier if someone with an actual HPC scheduler can test

[tcutts]

I wish I could - I can't build to test on my local system; I rapidly encounter all sorts of horrible gradle/proxy interaction problems behind the corporate firewall, so I can't do "make compile". Plus, I'm not a Java/Groovy programmer so I'm not sure about how to do the extra logic safely; the naive change is very simple, but would break nextflow on older versions of SLURM that lack the option.

Of course I could just vibe code the extra bits ... [ wince ]

[pditommaso]

The real challenge is that nextflow is not aware if this option is available in the running cluster. Make it the default is not an option because it would break all existing (legacy) clients

[tcutts]

Indeed, so I've now put in a PR (6579) with a patch for this which adds a configuration option for the executor, which defaults to false, the current behaviour. So the user has to explicitly enable it if they know their cluster supports it.

Patch has been tested on AstraZeneca's SLURM cluster (SLURM 24.11.5)