Skip to content

backoff strategy should be used for rate-limited errors on milvus or reducing batch_size config #54

Open
@filipecosta90

Description

@filipecosta90

It's recurrent to see the following type of errors on non-local setups:

pymilvus.exceptions.MilvusException: <MilvusException: (code=49, message=Retry run out of 10 retry times, message=request is rejected by grpc RateLimiter middleware, please retry later, req: /milvus.proto.milvus.MilvusService/Insert)>

Full traceback:

Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/ubuntu/vector-db-benchmark/engine/base_client/upload.py", line 90, in _upload_batch
    cls.upload_batch(ids, vectors, metadata)
  File "/home/ubuntu/vector-db-benchmark/engine/clients/milvus/upload.py", line 68, in upload_batch
    cls.upload_with_backoff(field_values, ids, vectors)
  File "/usr/local/lib/python3.10/dist-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/home/ubuntu/vector-db-benchmark/engine/clients/milvus/upload.py", line 75, in upload_with_backoff
    cls.collection.insert([ids, vectors] + field_values)
  File "/usr/local/lib/python3.10/dist-packages/pymilvus/orm/collection.py", line 443, in insert
    res = conn.batch_insert(self._name, entities, partition_name,
  File "/usr/local/lib/python3.10/dist-packages/pymilvus/decorators.py", line 109, in handler
    raise e
  File "/usr/local/lib/python3.10/dist-packages/pymilvus/decorators.py", line 105, in handler
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pymilvus/decorators.py", line 136, in handler
    ret = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pymilvus/decorators.py", line 80, in handler
    raise MilvusException(e.code, f"{timeout_msg}, message={e.message}") from e
pymilvus.exceptions.MilvusException: <MilvusException: (code=49, message=Retry run out of 10 retry times, message=request is rejected by grpc RateLimiter middleware, please retry later, req: /milvus.proto.milvus.MilvusService/Insert)>
"""


The above exception was the direct cause of the following exception:


Traceback (most recent call last):

  File "/home/ubuntu/vector-db-benchmark/run.py", line 79, in <module>
    app()

  File "/home/ubuntu/vector-db-benchmark/run.py", line 74, in run
    raise e

  File "/home/ubuntu/vector-db-benchmark/run.py", line 52, in run
    client.run_experiment(dataset, skip_upload, skip_search)

  File "/home/ubuntu/vector-db-benchmark/engine/base_client/client.py", line 70, in run_experiment
    upload_stats = self.uploader.upload(

  File "/home/ubuntu/vector-db-benchmark/engine/base_client/upload.py", line 56, in upload
    latencies = list(

  File "/usr/lib/python3.10/multiprocessing/pool.py", line 873, in next
    raise value

pymilvus.exceptions.MilvusException: <MilvusException: (code=49, message=Retry run out of 10 retry times, message=request is rejected by grpc RateLimiter middleware, please retry later, req: /milvus.proto.milvus.MilvusService/Insert)>

Given milvus configs dont specify the batch_size we're using 64 vectors, which seems to be constantly making the error state above.
I suggest to either respect API Rate Limits With a Backoff or reduce the batch size.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions