-
Notifications
You must be signed in to change notification settings - Fork 360
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add requests
argument to EnqueueLinksFunction
#1024
base: master
Are you sure you want to change the base?
Conversation
src/crawlee/_types.py
Outdated
|
||
It adds explicitly passed requests to the `RequestManager`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can imagine at least two instances of confusing behavior:
- you pass in
requests
andselector
, the function enqueues the requests (even though their elements don't match the selector), and it extracts and enqueues some additional links from the current page - you pass in
requests
andtransform_request_function
, but the function won't be called
I know that the docblock makes it pretty clear that this is what will happen, but I also know that there will be bug reports about this 😁
In my opinion, we should make two overloads and throw a runtime error if someone passes both requests
and one or more of the other arguments.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can imagine at least two instances of confusing behavior:
1. you pass in `requests` and `selector`, the function enqueues the requests (even though their elements don't match the selector), and it extracts and enqueues some additional links from the current page 2. you pass in `requests` and `transform_request_function`, but the function won't be called
I know that the docblock makes it pretty clear that this is what will happen, but I also know that there will be bug reports about this 😁
In my opinion, we should make two overloads and throw a runtime error if someone passes both
requests
and one or more of the other arguments.
Ok, so the function will basically become two in one alias for:
add_requests(requests)
add_requests(extract_requests(...))
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree with @janbuchar, rather be more strict here.
label: str | None = None, | ||
user_data: dict[str, Any] | None = None, | ||
transform_request_function: Callable[[RequestOptions], RequestOptions | RequestTransformAction] | None = None, | ||
requests: Sequence[str | Request] | None = None, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I now noticed that the JS counterpart accepts just urls
as an array of strings. We should either restrict this, or extend the JS version 🙂
If we choose restricting this one, then most of the other parameters (barring selector
) would actually start making sense in combination with urls
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would prefer to keep it as it is for consistency, since we use request: str | Request
everywhere else.
Description
Add
requests
argument toEnqueueLinksFunction
.Split
EnqueueLinksFunction
implementations toextract_links
andadd_requests
.Add overload variants of
EnqueueLinksFunction
.Raise error in
EnqueueLinksFunction
implementations if called with mutually exclusive arguments.Relates to : #906