fix!: Refactor service usage to rely on service_locator

[vdusek]

Description

• The service_container module has been completely refactored. It introduces changes to its usage, and resulting in many changes across the code base.
• While it remains possible to pass "services" directly to components as before, components now rely on the service_container internally. They no longer store service instances themselves.
• A new force flag has been added to the service_container's setters, which is especially useful for testing purposes.
• This also quite simplifies the memory_storage_client.
• We now have only set_storage_client, the same approach as for the event_manager. This is more flexible (allows more envs than just local & cloud). And in the SDK Actor, we can them based on the is_at_home.
• This is a breaking change, but it affects only the service_container interface.

Open discussion

• Should we go further and remove the option to pass configuration, event managers, or storage clients directly to components - requiring them to be set exclusively via the service_container? Thoughts are welcome.
  • No
• Better name for service_container? service_locator?
  • service_locator

Issues

• Closes: Streamline component management - unified access via service_container #699
• Closes: Crawler doesn't respect configuration argument #539
• Closes: How can I disable cache completely? #369
• It also unlocks the:
  • Fix the usage of Configuration class #670,
  • and maybe A typo(?) causing crashes in newer versions of apify + crawlee[parsel] apify-sdk-python#324 (comment).

Testing

• Existing tests, including those covering the service_container, have been updated to reflect these changes.
• New tests covering the MemoryStorageClient respects the Configuration.

Manual reproduction

• This code snippet demonstrates that the Crawler and MemoryStorageClient respects the custom Configuration.
• Note: Some fields remain non-working for now. These will be addressed in a future PR, as this refactor is already quite big. However, with the new architecture, those updates should now be easy.

import asyncio

from crawlee.configuration import Configuration
from crawlee.http_crawler import HttpCrawler, HttpCrawlingContext
from crawlee.service_container import set_configuration


async def main() -> None:
    config = Configuration(persist_storage=False, write_metadata=False)
    set_configuration(config)  # or Crawler(config=config)
    crawler = HttpCrawler()

    @crawler.router.default_handler
    async def request_handler(context: HttpCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url} ...')
        await context.push_data({'url': context.request.url})

    await crawler.run(['https://crawlee.dev/'])


if __name__ == '__main__':
    asyncio.run(main())

Checklist

• CI passed

[github-actions]

⚠️ Pull Request Tookit has failed!

Pull request is neither linked to an issue or epic nor labeled as adhoc!

[janbuchar]

Thanks for kicking this off!

[janbuchar]

We might want to try and resolve #369 here as well.

Additional context in Apify Slack - https://apify.slack.com/archives/C02JQSN79V4/p1723708212887119

[github-actions]

⚠️ Pull Request Tookit has failed!

Pull request is neither linked to an issue or epic nor labeled as adhoc!

[janbuchar]

Also apify/apify-sdk-python#324 (comment) is relevant here - if we could remove the need to initialize BasicCrawler under async with Actor, that would be great.

[Pijukatel]

Looks good to me, but maybe @janbuchar should approve as well.