[BUG] Discovery request queue is full

[benbierens]

Storage nodes in the testnet have began logging
Exception queueing discovery request topics="codex discoveryengine" tid=1 exc="AsyncQueue is full!"
Occurance is increasing over time.

This is very probably causing failure to discover blocks -> download slots -> start contracts.

I suspected the discovery worker had stopped processing the queue. But,
trace "Discovery task cancelled"
info "Exiting discovery task runner"
do not appear in the logs.

However, looking at proc discoveryTaskLoop it seems possible for the worker loop to be stopped without leaving any log entry by any exception that is not a CancelledError. (Should this be impossible due to compiler-checked exceptions?)

Can we confirm somehow whether the discovery worker has been stopped? (There should be 10 of them by default.) Or is there a likely explanation for why the workers can't keep up with the queue?

[benbierens]

Maybe this is a clue. When I fetch the logs to trace a storage contract, I see this one a lot:

ERR 2025-06-05 06:40:17.392+00:00 Sending of GetProviders message failed     topics="discv5" tid=1 error="GetProviders response message not received in time" count=1963468

By a lot, I mean more than 5500 times over a timespan of 20 minutes for a single storage node.

[benbierens]

Might this be downstream of long-blocking futures? If a node is tied up generating proofs for several seconds, could that cause these failures on other nodes?

[gmega]

I'll take this one, I might actually be able to fix it.