-
Notifications
You must be signed in to change notification settings - Fork 13
Description
Context
Default RequestQueue
offers many features apart from request managment, like the possibility of multiple producers/consumers working with it, real-time persistence, and integration with console. To support all of it, it comes with some costs - actual financial costs to use request queue and increased network traffic as the request queue is sending many API calls related to request handling and locking. Some use cases do not need those additional features of the request queue and thus are only affected by the costs with no benefit.
Proposed solution
-
Add single consumer variant of the
RequestQueue
- this will drop the locking mechanism of the RequestQueue and reduce costs and network traffic, while preserving all other benefits ofRequestQueue
-
Add new specialized
RequestQueue
client that will not be based on platform-backedRequestQueue
to even further reduce the costs. Possibly just extendMemoryRequestQueueClient
with persistence capabilities to KVS. Use case is single consumer/producer request queue with on-demand persistence that is massively cheaper compared to platform-backedRequestQueue
. - For example this POC -
Update documentation and describe all the variants of
RequestQueue
, their benefits, drawbacks, and recommended use cases. Also mention crawlee-basedRequestQueue
clients that still have some niche use case even on Apify platform.