Add JobSet Support to PodGrouper

[rich7420]

What you would like to be added?

We'd like to add a PodGrouper plugin for JobSet workloads so it can automatically create PodGroups for gang scheduling. The plugin will create a PodGroup for each Job replica in a JobSet, using the naming pattern pg-<jobset-name>-<replicatedjob-name>-<job-index>-<jobset-uid>. It sets MinAvailable based on spec.replicatedJobs[].template.spec.parallelism. The implementation follows the same pattern as our existing plugins and comes with unit tests.

Why is this needed?

JobSet is pretty commonly used for distributed training workloads, but right now KAI Scheduler's PodGrouper doesn't support it. This means users have to manually create PodGroups or use the default grouper, which doesn't handle JobSet's gang scheduling needs well. With this feature, we can enable automatic gang scheduling for JobSet workloads and avoid sequencing deadlocks by creating separate PodGroups per Job replica. This way users can take advantage of KAI Scheduler's gang scheduling without having to manage PodGroups manually.

cc @romanbaron

[davidLif]

Hi @rich7420 ,

I think that the JobSet definition have a few features we need to consider:
1-
startupPolicyOrder: InOrder. In this case, each replicatedJob executes separatly. We might want to create a separate podgroup for each replicatedJob in this case.

2-
If you look at https://github.com/kubernetes-sigs/jobset?tab=readme-ov-file#conceptual-diagram and read about the replicas field in a replicatedJob object, you can sse that the MinAvailable calculation needs to take the replicas field too as a part of the calculation.

3-
I think we should discuss the what should the MinAvailable value be when in cases where parallelism != completions.
For example, if completions = parallelism + 1 and parallelism > 1 and 2 pods reached a "complete" state, what should be the MinAvailable value? This is a valid state, but the number of running pods might be lower then parallelism. This will trigger a stale gang eviction.

[rich7420]

@davidLif Thanks for the suggestions!
I'll follow the clues and have a PR as soon as possible!