Connection breaks when scaling Triton servers deployed via interLink

[kondratyevd]

When Triton servers are deployed on interLink virtual nodes (in Slurm jobs), a single-server setup works fine. However, when number of clients is high enough to trigger autoscaling, the client-server connections break and clients fail.

My main suspicion is that servers started on interLink nodes take too long to load (time to establish wstunnel, pulling singularity image, maybe something else).

Possible ways to fix:

• tweak readiness probes for Triton servers
• would be even better if Triton server itself would appear ready a bit later, eliminating a need to fine-tune Kubernetes probes
• currently it's inconvenient to debug - maybe a better aggregation of logs would help

[kondratyevd]

The error on the client side is this:

Thread [0] had error: upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: 111     

[kondratyevd]

The issue seems to be due to startupProbe being ignored:

• if I set startupProbe.initialDelaySeconds to a high value (300s), Triton pods appear "Ready" and "Running" pretty fast, definitely not waiting for 300s.
• in SuperSONIC deployment without interLink with an identical startupProbe configuration, the Triton pod appears "Running", but not "Ready" - this is the expected behavior.

Note: this was tested with interLink Slurm plugin version 0.5.2