[Feat][kubectl-plugin] Config file for creating RayClusters

[MortalHappiness]

As we support more use-cases for cluster creation, it won’t scale to add CLI flags for every possible permutation of RayClusters. We should consider supporting config files (in YAML) to capture high-level parameters for cluster creation. The challenge here is designing config files that are significantly simpler than writing RayCluster YAML directly without sacrificing too much utility. One consideration would be to use a similar format as the existing values.yaml for the Ray Cluster helm chart.

Possible Config

context: CURRENT_CONTEXT
namespace: [none]

labels: {}
annotations: [}

ray-version: CURRENT_VERSION
image: rayproject/ray:CURRENT_VERSION

head-cpu: 2
head-memory: 4Gi
head-gpu: 0
head-ray-start-params: {}

worker-groups:
- name: worker
  worker-replicas: 1
  worker-cpu: 2
  worker-memory: 4Gi
  worker-gpu: 0
  worker-ray-start-params: {}

dry-run: false
wait: false
timeout: 5m

gke:
  # GCS bucket volume mount
  gcsfuse:
    bucket0name: my-bucket
    # optional mount options
    mount-options: implicit-dirs,uid=1000,gid=100
    # optional mount path that defaults to /mnt/gcs-data
    # the bucket will be mounted at /mnt/gcs-data on both head and worker nodes
    mount-path: /mnt/gcs-data
    # optional resource configs for gcsfuse sidecar container
    resources:
      cpus: 250m
      memory: 256Mi
      ephemeral-storage: 5Gi

eks: {}

Example command

kubectl ray create cluster (NAME) --cluster-config <path-to-config>

Discussion

Isn't this somewhat redundant with helm install raycluster? After all, helm template can achieve a similar outcome.

[kevin85421]

/assign @davidxia

[davidxia]

looks like the assignment command isn't working?

[davidxia]

I've also updated the schema based on preliminary feedback in #3225.

[davidxia]

Isn't this somewhat redundant with helm install raycluster? After all, helm template can achieve a similar outcome.

I just saw this. Here's my initial thoughts.

• Some users may not be able to or want to use helm. At Spotify Helm usage isn't well supported or common at Spotify.
• Helm might not abstract K8s away enough for some users. The RayCluster K8s YAML is still created and used.
• We can create integrations for and abstract away more things like GCS Fuse (see the example in the PR description) that helm cannot. With helm, users still have to understand all the key-values they need to set with helm --set key=value which requires both an understanding of where to change RayCluster YAML and finding the right helm key to set.

[kevin85421]

@davidxia I can only assign the issue to you when you comment on the issue haha