Faster shard scaling

[rdettai]

Description

Indexers return a lot of 429 when ingest rates fluctuate, even if the overall indexing capacity (~5MB/core) is note reached. This PR makes some small parameter adjustments to decrease the overall error rates on typical workloads. It will be the topic for another issue/PR to introduce the structual changes that are necessary to improve the routing strategy.

• Customizable shard burst limit with default increased to 50MB: helps absorbing the load until the next scale up
• Shard scaling factor: helps keeping up with load fluctuations on workloads with high number of shards (e.g having +20MB/s on an index where the average ingest rate is 100MB/s is more likely than +20MB/s on an index where the average rate is 1MB/s)

Related to #5270

How was this PR tested?

Adhoc python load testing framework: https://github.com/rdettai/qw-ingest-tests

[rdettai]

Test setup:

• single node cluster
• apply a continuous load on a new index
• correlate shard counts with 429 throttling

Observations:

• there is a high latency between a scaling action and the entry being added to the router (5+s)
  • shard_burst_limit helps absorbing part of this delay, but even with a high burst limit (100MB), a fairly low ingest rate (20MB/s) runs into throttling before the router learns about the new shards
  • when scaling, in the time interval where the new shards are created but not yet added to the router, the load factor calculated is artificially lowered: the new shards are already taken into account in the capacity calculation but not for the load calculation (the requests that could have gone to these shards are still rejected with 429). This sometimes delays the next scale up action.

[fulmicoton-dd]

I'm not sure about the floor? I would have expected ceil here.

[rdettai]

If it was a ceil, it would mean that scaling up by this number of shards would get us slightly below the scale_up_shards_long_term_threshold_mib_per_sec. For small shard counts it might get us a bit to close to scale_down_shards_threshold_mib_per_sec