failed_request_handler runs and logs but context.push_data(...) does not write to dataset (PlaywrightCrawler)

[loic-bellinger]

When a request fails in PlaywrightCrawler, my @crawler.failed_request_handler runs (I can see context.log output) but await context.push_data(...) does not create any dataset rows. Logging works (I can see the context.log.info(...) lines), only the dataset push appears to be ignored.

Minimal repro attached below.

import asyncio

from crawlee.crawlers import (
    PlaywrightCrawler,
    PlaywrightCrawlingContext,
    BasicCrawlingContext,
)


async def main() -> None:
    crawler = PlaywrightCrawler(
        max_requests_per_crawl=10,
        max_request_retries=2,
        headless=True,
        browser_type='chromium',
    )

    @crawler.router.default_handler
    async def request_handler(context: PlaywrightCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url} ...')

    @crawler.failed_request_handler
    async def failed_request_handler(context: BasicCrawlingContext, error: Exception) -> None:
        context.log.info(f'failed_request_handler: processing {context.request.url} ...')
        await context.push_data(
            dataset_name="failed_request_handler_errors",
            data={
                "failed_url": context.request.url,
                "label": context.request.label,
                "error_type": type(error).__name__,
                "error_message": str(error),
                "retry_count": context.request.retry_count,
                "status": "failed",
            },
        )

    await crawler.run(['https://www.info.gouv.fr/totalnonsense'])


if __name__ == '__main__':
    asyncio.run(main())

I'm using crawlee[playwright]>=1.0.4

Relevant part of the documentation is https://crawlee.dev/python/docs/guides/request-router#failed-request-handler

[janbuchar]

Hi @loic-bellinger, thanks for reporting this! The issue is that we "buffer" any storage-related calls on the context object so that partial results of failing request handler runs are not stored. This is an unintended side effect of this feature. We'll do our best to fix this soon 🙂

As a temporary workaround, you may try this:

dataset = await Dataset.open(name="failed_request_handler_errors")
await dataset.push_data({...})

It should bypass the "buffering" mechanism.

[loic-bellinger]

Wouldn't calling dataset = await Dataset.open(name="failed_request_handler_errors") on each error create some overhead?

[janbuchar]

Wouldn't calling dataset = await Dataset.open(name="failed_request_handler_errors") on each error create some overhead?

There is a lot of caching, you should be fine. But let us know if anything happens!