feat: Implement Scrapy HTTP cache backend #403
Conversation
honzajavorek
changed the title
honzajavorek
had a problem deploying
to
fork-pr-integration-tests
GitHub Actions
Failure
vdusek
added
t-tooling
|
Thanks for the review! I considered this to be a kick off which will definitely get a review with changes requested. If #404 lets me, I'll work on addressing the comments in the upcoming days. |
|
I bet I've seen comments about adding this to Apify settings by default, but I can't see them now. I think it's a good idea to set the cache storage by default, but I wouldn't turn on caching by default, because it's application-specific how caching should work exactly (e.g. expiration time). We should just document somewhere that this is a preferred solution to handling the forced restarts, if they happen. |
Agreed, thanks 🙂. |
|
Just for the record, the comment I've seen didn't get lost, it's the final comment of the review 😄 |
This code has been originally developed for the https://github.com/juniorguru/plucker/ project, which is licensed under AGPL-3.0-only. I am the sole author of that code and hereby I grant the https://github.com/apify/apify-sdk-python/ project the right to use it under an Apache-2.0 license, without the copyleft taking effect. My intention is to contribute this code to the upstream, remove it from my project, and only import the component from the apify package as a dependency, as I believe this component could be useful to other users of the apify package.
This error was introduced when I tried to make all the linters happy, uh.
honzajavorek
force-pushed
the
honzajavorek/cache
branch
from
92f6435 to
3a2cc13
Compare
|
I think I'm done for now! |
I successfully use this code in my project. In 381c044 I explicitly specify licensing. I didn't add docs or anything else (yet), just made sure the code passes linters and a type check.
Relates: apify/actor-templates#303