Forward cross chain messages for different recipients actually in parallel

[ndr-ds]

Motivation

Right now we're using FuturesUnordered, which will poll the futures concurrently in the same task.

Proposal

Use JoinSet so we actually can poll these in parallel, using different cores

Test Plan

Benchmarked a network without this, and with this, and with this our cross chain message queue wait time improved by about 10%. This was on a network with 4 worker threads per shard.

Release Plan

• Nothing to do / These changes follow the usual release cycle.

[ndr-ds]

• Forward cross chain messages for different recipients actually in parallel #5005 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[deuszx]

In the title you say it's for different recipients but I fail to see where it's guaranteed in this code. Can you point it to me?

[ndr-ds]

Wether we initially spawn a task or not is controlled by job_states, which is a HashMap from QueueId to JobState. So stuff that has the same sender and recipient will use the same task.

[ma2bd]

Interesting. What is the CPU-heavy operation taking advantage of multiple cores?

[ndr-ds]

Like I mentioned in the description, this wasn't really a bottleneck. Was just a place that I thought could be one. But the improvements were small, so we're probably not even exhausting the 1 core with this, as it just gets slightly faster in parallel.
I'm also open to closing this, if people think it's not worth it :)
I personally think it doesn't hurt 🤷🏻‍♂️ but up to you guys

[ma2bd]

I found the answer to my question: it's not about CPU. It's about the select! branches preventing other tasks from being polled.