[Prometheus] Add ray_cluster_provisioned_duration_seconds metric

[win5923]

Why are these changes needed?

Add ray_cluster_provisioned_duration_seconds metric to track the time from RayClusters created to all ray pods are ready for the first time (RayClusterProvisioned).

Manual test:

$ k apply -f config/samples/ray-cluster.sample.yaml

$ echo $(( $(date -d "$(kubectl get raycluster raycluster-kuberay -o=jsonpath='{.status.conditions[?(@.type=="HeadPodReady")].lastTransitionTime}')" +%s) - $(date -d "$(kubectl get raycluster raycluster-kuberay -o=jsonpath='{.metadata.creationTimestamp}')" +%s) )) "seconds"
41 seconds

$ k apply -f config/samples/ray-cluster.embed-grafana.yaml

$ echo $(( $(date -d "$(kubectl get raycluster raycluster-embed-grafana -o=jsonpath='{.status.conditions[?(@.type=="HeadPodReady")].lastTransitionTime}')" +%s) - $(date -d "$(kubectl get raycluster raycluster-embed-grafana -o=jsonpath='{.metadata.creationTimestamp}')" +%s) )) "seconds"
16 seconds

$ k apply -f config/samples/ray-cluster.embed-grafana.yaml -n test

$ echo $(( $(date -d "$(kubectl get raycluster raycluster-embed-grafana -n test -o=jsonpath='{.status.conditions[?(@.type=="HeadPodReady")].lastTransitionTime}')" +%s) - $(date -d "$(kubectl get raycluster raycluster-embed-grafana -n test -o=jsonpath='{.metadata.creationTimestamp}')" +%s) )) "seconds"
15 seconds

Related issue number

Closes #3172

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[win5923]

@troychiu PTAL, thx.

[win5923]

Not sure what bucket ranges would be suitable for most users.

[kevin85421]

cc @troychiu let's prioritize reviewing this PR.

[win5923]

This PR is a follow-up to #3310.

[kevin85421]

The metric should not be recorded in calculateStatus. It should only be recorded when the status update succeeds, to avoid counting it more than once.