kubernetes: replace probe's InitialDelay with StartupProbe

[z0rc]

Readiness and liveness probes don't run until the startup probe succeeds, so a short startup period gets the pod ready shortly after buildkitd starts instead of waiting for probePeriodSeconds.

In my environment this brings time required for buildkitd container to become ready from ~31 seconds to ~6 seconds.

[crazy-max]

Could you open an issue with a minimal reproducer for the ~31s to ~6s improvement and link it here? Please include the Kubernetes, Buildx, and BuildKit versions, the exact builder creation/bootstrap commands and configuration, and timestamped BuildKit logs and pod status showing container start, daemon readiness, and Pod Ready before and after this change.

Readiness probes can run outside their configured interval while the container is unready, so we'd like to establish where the delay comes from in your environment. Please also explain the choice of 24 startup failures with a 60s timeout per attempt, since that substantially extends the time before a hung startup is restarted.

[z0rc]

@crazy-max see #4125 for full writeup.

Please also explain the choice of 24 startup failures with a 60s timeout per attempt, since that substantially extends the time before a hung startup is restarted.

I just reused value from other probes here. I can reduce this value to 5 seconds only for startup probe. See measurements in linked ticket.