Skip to content

Add specialized/parametrized RequestQueueClients #513

@Pijukatel

Description

@Pijukatel

Context
Default RequestQueue offers many features apart from request managment, like the possibility of multiple producers/consumers working with it, real-time persistence, and integration with console. To support all of it, it comes with some costs - actual financial costs to use request queue and increased network traffic as the request queue is sending many API calls related to request handling and locking. Some use cases do not need those additional features of the request queue and thus are only affected by the costs with no benefit.

Proposed solution

  • Add single consumer variant of the RequestQueue - this will drop the locking mechanism of the RequestQueue and reduce costs and network traffic, while preserving all other benefits of RequestQueue

  • Add new specialized RequestQueue client that will not be based on platform-backed RequestQueue to even further reduce the costs. Possibly just extend MemoryRequestQueueClient with persistence capabilities to KVS. Use case is single consumer/producer request queue with on-demand persistence that is massively cheaper compared to platform-backed RequestQueue. - For example this POC

  • Update documentation and describe all the variants of RequestQueue, their benefits, drawbacks, and recommended use cases. Also mention crawlee-based RequestQueue clients that still have some niche use case even on Apify platform.

Metadata

Metadata

Assignees

Labels

documentationImprovements or additions to documentation.enhancementNew feature or request.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions