Description
Metrics hot path uses RWLock to sync access to HashMap. Though there is no competition to acquire Read lock in steady state, this is still a cause of contention.
This is easily visible in the stress tests, where throughput is 1/5th that of Otel .NET which uses a comparable stress test. The single thread performance between Otel Rust and .NET is comparable.
Based on couple of experiments, this can be solved via either leveraging a ConcurrentHashMap (that does not require read locks), or sharding. Otel .NET, Java solves this via ConcurrentHashMap, but as of this writing, Rust std library does not have one.
Opening an issue to track improving this. The final decision about sharding vs using external crates vs simply-wait-until-rust-std-adds-concurrenthashmap can be discussed in this issue.