Enable persistent Ray cluster state via external Redis GCS fault tolerance

[kramaranya]

Issue link

RHOAIENG-11115

What changes have been made

Provided Ray cluster head pod persistency through GCS fault tolerance.
Added new config options: enable_gcs_ft, redis_address, redis_password_secret and external_storage_namespace

Verification steps

• Provision a cluster
• Deploy Redis in your cluster ( you can use this example)
• Deploy KubeRay operator (1.3.0 or newer, RHOAI currently provides an older version)
• Create RayCluster with GCS fault tolerance enabled, for example:

cluster = Cluster(ClusterConfiguration(
        name='raycluster',
        head_cpu_requests='500m',
        head_cpu_limits='500m',
        head_memory_requests=2,
        head_memory_limits=2,
        num_workers=1,
        worker_cpu_requests='250m',
        worker_cpu_limits=1,
        worker_memory_requests=4,
        worker_memory_limits=4,
        enable_gcs_ft=True,
        redis_address="redis:6379",
        redis_password_secret={
            "name": "redis-password-secret",
            "key": "password"
        }
    ))

• Make sure all pods are running
• Create a detached actor
• Check if the actor is visible - ray list actors from RayCluster head pod
• Restart the head pod
• Run ray list actors - you should see previously created actor

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • Testing is not required for this change

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign bobbins228 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment