Better LLM retry behavior

[rbren]

End-user friendly description of the problem this fixes or functionality that this introduces

• Include this change in the Release Notes. If checked, you must provide an end-user friendly description for your change below
  no changelog

---

Give a summary of what the PR does, explaining any non-trivial design decisions

The LLM is retrying a lot of unrecoverable exceptions, which makes it look like the app is just stuck.

The current configuration also waits a total of 11 minutes (!) for a good response, not including the request time, which can add ~5-8 minutes to that total. So the app looks VERY stuck.

We could potentially move this into a config if these errors are common enough that eval needs them. CC @xingyaoww

---

Link of any specific issues this addresses

---

To run this PR locally, use the following command:

docker run -it --rm   -p 3000:3000   -v /var/run/docker.sock:/var/run/docker.sock   --add-host host.docker.internal:host-gateway   -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:185288b-nikolaik   --name openhands-app-185288b   docker.all-hands.dev/all-hands-ai/openhands:185288b

[enyst]

503 is a transitory error, we could probably keep it?

[rbren]

Hmm. It's transitory but also unexpected...

I'm open to it but I lean towards telling the user their LLM is flaking out rather than OpenHands looking like it's slow

[enyst]

I kinda agree with you actually. We always had a problem in understanding our retry settings, because it's a bit weird to figure out a sensible default for "unexpected stuff happened".

And now we do allow the user to continue normally after reporting the error.

eval is the exception, I'd love to hear from Xingyao on that.

[enyst]

There are some issues on litellm on this, the exceptions as defined are mixing permanent and transitory exceptions from the provider. We have some weird code due to that. I would agree that cleaning them and start again is reasonable. 😅

[enyst]

Small related detail, there's a try/except due to retries in llm.py, which is unnecessary even in main, and more so now. We might as well clean that out:

• Chore: clean up LLM (prompt caching, supports fn calling), leftover renames #6095

[enyst]

Please see also a small follow-up here:

• [rbren no-retries] add user-friendly messages #6576

[rbren]

Thanks @enyst! Any lingering issues here?

[enyst]

I think it would be great if @xingyaoww can take a look, because it's possible that the removed exceptions are happening.

Up to you.