Move waiting logic for host orchestrator settlement to backend

[k311093]

While interacting with host orchestrator right after creating host
API is returned, there is a possibility of getting 503 service
temporarily unavailable error because host it created and booted but
host orchestrator is not executed / ready yet.

Handling logic for waiting host orchestrator readiness is currently
in the client logic. Moving this logic into cloud orchestrator API
endpoint would make the client that uses host chestrator API simpler.

Context: b/501288123

[ser-io]

Adding an inside logic to wait for when the host is ready to interact with shouldn't be part of the CreateHost request logic on the backend side.

CreateHost returns success when the host is created and host resources are allocated, not when the host is ready to interact with. To do this properly, you'd need to add and implement a new http endpoint that tests whether the host is ready, still clients would need to make these calls after CreateHost returned in order to determine whether the host is ready to interact with. However, it's up to every client implementation how they want to handle this, they can always keep doing what the current client is doing.

[jemoreira]

Adding an inside logic to wait for when the host is ready to interact with shouldn't be part of the CreateHost request logic on the backend side.

CreateHost returns success when the host is created and host resources are allocated, not when the host is ready to interact with.

What use is a host that doesn't have a host orchestrator? If the answer is "none", then it would make sense to only mark the operation to create a host as completed once the host orchestrator is ready. It simplifies the api and puts the need for waiting in a single place (the cloud orchestrator) rather than in every client. The CO is already waiting for the host to be created, it might as well wait a bit longer.

[jemoreira]

Context: b/501288123

Please add more details to the description and the title of the PR.

What is "host orchestrator settlement"?

Which backend? The HO or the CO or something else?

Why is this change useful/needed? What are the pros and cons? Even if you write a description in the bug that's not publicly available like this PR is.

[k311093]

Context: b/501288123

Please add more details to the description and the title of the PR.

What is "host orchestrator settlement"?

Which backend? The HO or the CO or something else?

Why is this change useful/needed? What are the pros and cons? Even if you write a description in the bug that's not publicly available like this PR is.

Updated commit message with descriptions.

[0405ysj]

What use is a host that doesn't have a host orchestrator? If the answer is "none"

+1. I expect users are understanding the meaning of host as a host environment to launch Cuttlefish instances, and the route of orchestrating CF instances is Host Orchestrator. I think health of Host Orchestrator is more representative than health of computing unit(e.g. GCE or docker instance).

From API view, I expect users create a host and verify with 3 steps; POST /v1/zones/{zone}/hosts, POST /v1/zones/{zone}/operations/{operation}/:wait, GET /v1/zones/{zone}/hosts/{host}/. Invoking those endpoints should be responsible by clients though. But having another round of wait by setting retry count for requesting GET /v1/zones/{zone}/hosts/{host}/ would be unexpected behavior in users' perspective, I think.

[k311093]

Hi @jemoreira can you please re-review this PR since request change is not closed yet.

Cannot re-request review for clear change request. Looks like comment while change request is resolved.