Open
Description
It's recurrent to see the following type of errors on non-local setups:
pymilvus.exceptions.MilvusException: <MilvusException: (code=49, message=Retry run out of 10 retry times, message=request is rejected by grpc RateLimiter middleware, please retry later, req: /milvus.proto.milvus.MilvusService/Insert)>
Full traceback:
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/ubuntu/vector-db-benchmark/engine/base_client/upload.py", line 90, in _upload_batch
cls.upload_batch(ids, vectors, metadata)
File "/home/ubuntu/vector-db-benchmark/engine/clients/milvus/upload.py", line 68, in upload_batch
cls.upload_with_backoff(field_values, ids, vectors)
File "/usr/local/lib/python3.10/dist-packages/backoff/_sync.py", line 105, in retry
ret = target(*args, **kwargs)
File "/home/ubuntu/vector-db-benchmark/engine/clients/milvus/upload.py", line 75, in upload_with_backoff
cls.collection.insert([ids, vectors] + field_values)
File "/usr/local/lib/python3.10/dist-packages/pymilvus/orm/collection.py", line 443, in insert
res = conn.batch_insert(self._name, entities, partition_name,
File "/usr/local/lib/python3.10/dist-packages/pymilvus/decorators.py", line 109, in handler
raise e
File "/usr/local/lib/python3.10/dist-packages/pymilvus/decorators.py", line 105, in handler
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pymilvus/decorators.py", line 136, in handler
ret = func(self, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pymilvus/decorators.py", line 80, in handler
raise MilvusException(e.code, f"{timeout_msg}, message={e.message}") from e
pymilvus.exceptions.MilvusException: <MilvusException: (code=49, message=Retry run out of 10 retry times, message=request is rejected by grpc RateLimiter middleware, please retry later, req: /milvus.proto.milvus.MilvusService/Insert)>
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/ubuntu/vector-db-benchmark/run.py", line 79, in <module>
app()
File "/home/ubuntu/vector-db-benchmark/run.py", line 74, in run
raise e
File "/home/ubuntu/vector-db-benchmark/run.py", line 52, in run
client.run_experiment(dataset, skip_upload, skip_search)
File "/home/ubuntu/vector-db-benchmark/engine/base_client/client.py", line 70, in run_experiment
upload_stats = self.uploader.upload(
File "/home/ubuntu/vector-db-benchmark/engine/base_client/upload.py", line 56, in upload
latencies = list(
File "/usr/lib/python3.10/multiprocessing/pool.py", line 873, in next
raise value
pymilvus.exceptions.MilvusException: <MilvusException: (code=49, message=Retry run out of 10 retry times, message=request is rejected by grpc RateLimiter middleware, please retry later, req: /milvus.proto.milvus.MilvusService/Insert)>
Given milvus configs dont specify the batch_size
we're using 64 vectors, which seems to be constantly making the error state above.
I suggest to either respect API Rate Limits With a Backoff or reduce the batch size.
Metadata
Metadata
Assignees
Labels
No labels