Skip to content

Logs from browserforge #423

New issue
New issue
Closed
Closed
Logs from browserforge#423
Assignees
Pijukatel
Labels
bugSomething isn't working.t-toolingIssues with this label are in the ownership of the tooling team.

Description

@vdusek
vdusek
opened on Mar 4, 2025

Before

  • Before browserforge (Crawlee < 0.6)
2025-03-04T09:35:42.108Z ACTOR: Pulling Docker image of build vbJ3dgYm28POtiN1j from repository.
2025-03-04T09:35:42.311Z ACTOR: Creating Docker container.
2025-03-04T09:35:42.358Z ACTOR: Starting Docker container.
2025-03-04T09:35:44.092Z [apify] INFO  Initializing Actor...
2025-03-04T09:35:44.094Z [apify] INFO  System info ({"apify_sdk_version": "2.3.1", "apify_client_version": "1.9.2", "crawlee_version": "0.6.1", "python_version": "3.13.2", "os": "linux"})
2025-03-04T09:35:44.119Z [apify] DEBUG Debug message
2025-03-04T09:35:44.123Z [apify] INFO  Info message
2025-03-04T09:35:44.124Z [apify] WARN  Warning message
2025-03-04T09:35:44.126Z [apify] ERROR Error message
2025-03-04T09:35:44.128Z [apify] ERROR Exception message
2025-03-04T09:35:44.131Z       Traceback (most recent call last):
2025-03-04T09:35:44.133Z         File "/usr/src/app/src/main.py", line 25, in main
2025-03-04T09:35:44.134Z           raise ValueError('Dummy ValueError')
2025-03-04T09:35:44.136Z       ValueError: Dummy ValueError
2025-03-04T09:35:44.139Z [apify] INFO  Multi
2025-03-04T09:35:44.141Z line
2025-03-04T09:35:44.142Z log
2025-03-04T09:35:44.144Z message
2025-03-04T09:35:44.146Z [apify] ERROR Actor failed with an exception
2025-03-04T09:35:44.147Z       Traceback (most recent call last):
2025-03-04T09:35:44.149Z         File "/usr/src/app/src/main.py", line 33, in main
2025-03-04T09:35:44.150Z           raise RuntimeError('Dummy RuntimeError')
2025-03-04T09:35:44.152Z       RuntimeError: Dummy RuntimeError
2025-03-04T09:35:44.154Z [apify] INFO  Exiting Actor ({"exit_code": 91})

After

  • With browserforge (Crawlee ~= 0.6)
2025-03-04T09:35:42.108Z ACTOR: Pulling Docker image of build vbJ3dgYm28POtiN1j from repository.
2025-03-04T09:35:42.311Z ACTOR: Creating Docker container.
2025-03-04T09:35:42.358Z ACTOR: Starting Docker container.
2025-03-04T09:35:43.463Z Downloading model definition files...
2025-03-04T09:35:43.725Z input-network.zip             OK!
2025-03-04T09:35:43.727Z browser-helper-file.json      OK!
2025-03-04T09:35:43.729Z header-network.zip            OK!
2025-03-04T09:35:43.731Z headers-order.json            OK!
2025-03-04T09:35:43.732Z fingerprint-network.zip       OK!
2025-03-04T09:35:44.092Z [apify] INFO  Initializing Actor...
2025-03-04T09:35:44.094Z [apify] INFO  System info ({"apify_sdk_version": "2.3.1", "apify_client_version": "1.9.2", "crawlee_version": "0.6.1", "python_version": "3.13.2", "os": "linux"})
2025-03-04T09:35:44.119Z [apify] DEBUG Debug message
2025-03-04T09:35:44.123Z [apify] INFO  Info message
2025-03-04T09:35:44.124Z [apify] WARN  Warning message
2025-03-04T09:35:44.126Z [apify] ERROR Error message
2025-03-04T09:35:44.128Z [apify] ERROR Exception message
2025-03-04T09:35:44.131Z       Traceback (most recent call last):
2025-03-04T09:35:44.133Z         File "/usr/src/app/src/main.py", line 25, in main
2025-03-04T09:35:44.134Z           raise ValueError('Dummy ValueError')
2025-03-04T09:35:44.136Z       ValueError: Dummy ValueError
2025-03-04T09:35:44.139Z [apify] INFO  Multi
2025-03-04T09:35:44.141Z line
2025-03-04T09:35:44.142Z log
2025-03-04T09:35:44.144Z message
2025-03-04T09:35:44.146Z [apify] ERROR Actor failed with an exception
2025-03-04T09:35:44.147Z       Traceback (most recent call last):
2025-03-04T09:35:44.149Z         File "/usr/src/app/src/main.py", line 33, in main
2025-03-04T09:35:44.150Z           raise RuntimeError('Dummy RuntimeError')
2025-03-04T09:35:44.152Z       RuntimeError: Dummy RuntimeError
2025-03-04T09:35:44.154Z [apify] INFO  Exiting Actor ({"exit_code": 91})

Result

  • As a result test test_actor_logging is skipped.

Possible solutions

  • Remove the browserforge output and rm skip flag from test_actor_logging.
  • Leave it as it is and rm (or update) the test_actor_logging test.
    • in case of updating - there is no line order guaranteed.

Activity

  1. github-actions
    added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Mar 4, 2025
  2. Pijukatel
    self-assigned this
    on Mar 4, 2025

  3. vdusek commented on Mar 4, 2025

    @vdusek
    vdusek
    on Mar 4, 2025
    ContributorAuthor

    It is important to note that during every Actor run, zip files are now downloaded from the apify/fingerprint-suite repository.

  4. Erol444 commented on Mar 8, 2025

    @Erol444
    Erol444
    on Mar 8, 2025

    Not sure if related, but I get this error every time I run an actor:

    ACTOR: Pulling Docker image of build {MY_ID} from repository.
2025-03-08T12:46:21.303Z ACTOR: Creating Docker container.
2025-03-08T12:46:21.387Z ACTOR: Starting Docker container.
2025-03-08T12:46:23.361Z Downloading model definition files...
2025-03-08T12:46:23.615Z Error downloading fingerprint-network.zip: [Errno 13] Permission denied: '/usr/local/lib/python3.13/site-packages/browserforge/fingerprints/data/fingerprint-network.zip'
2025-03-08T12:46:23.618Z Error downloading fingerprint-network.zip: [Errno 13] Permission denied: '/usr/local/lib/python3.13/site-packages/browserforge/fingerprints/data/fingerprint-network.zip'
2025-03-08T12:46:23.620Z Error downloading fingerprint-network.zip: [Errno 13] Permission denied: '/usr/local/lib/python3.13/site-packages/browserforge/fingerprints/data/fingerprint-network.zip'
2025-03-08T12:46:23.622Z Error downloading fingerprint-network.zip: [Errno 13] Permission denied: '/usr/local/lib/python3.13/site-packages/browserforge/fingerprints/data/fingerprint-network.zip'
2025-03-08T12:46:23.625Z Error downloading fingerprint-network.zip: [Errno 13] Permission denied: '/usr/local/lib/python3.13/site-packages/browserforge/fingerprints/data/fingerprint-network.zip'
2025-03-08T12:46:23.627Z Downloading model definition files...
2025-03-08T12:46:23.645Z Error downloading input-network.zip: [Errno 13] Permission denied: '/usr/local/lib/python3.13/site-packages/browserforge/headers/data/input-network.zip'
2025-03-08T12:46:23.648Z Error downloading input-network.zip: [Errno 13] Permission denied: '/usr/local/lib/python3.13/site-packages/browserforge/headers/data/input-network.zip'
2025-03-08T12:46:23.650Z Error downloading input-network.zip: [Errno 13] Permission denied: '/usr/local/lib/python3.13/site-packages/browserforge/headers/data/input-network.zip'
2025-03-08T12:46:23.652Z Error downloading input-network.zip: [Errno 13] Permission denied: '/usr/local/lib/python3.13/site-packages/browserforge/headers/data/input-network.zip'
2025-03-08T12:46:23.658Z Traceback (most recent call last):
2025-03-08T12:46:23.667Z   File "<frozen runpy>", line 198, in _run_module_as_main
2025-03-08T12:46:23.669Z   File "<frozen runpy>", line 88, in _run_code
2025-03-08T12:46:23.671Z   File "/usr/src/app/src/__main__.py", line 3, in <module>
2025-03-08T12:46:23.674Z     from .main import main
2025-03-08T12:46:23.676Z   File "/usr/src/app/src/main.py", line 10, in <module>
2025-03-08T12:46:23.678Z     from apify import Actor, Request
2025-03-08T12:46:23.680Z   File "/usr/local/lib/python3.13/site-packages/apify/__init__.py", line 15, in <module>
2025-03-08T12:46:23.682Z     from apify._actor import Actor
2025-03-08T12:46:23.684Z   File "/usr/local/lib/python3.13/site-packages/apify/_actor.py", line 28, in <module>
2025-03-08T12:46:23.686Z     from apify._charging import ChargeResult, ChargingManager, ChargingManagerImplementation
2025-03-08T12:46:23.689Z   File "/usr/local/lib/python3.13/site-packages/apify/_charging.py", line 17, in <module>
2025-03-08T12:46:23.692Z     from apify.storages import Dataset
2025-03-08T12:46:23.694Z   File "/usr/local/lib/python3.13/site-packages/apify/storages/__init__.py", line 3, in <module>
2025-03-08T12:46:23.698Z     from ._request_list import RequestList
2025-03-08T12:46:23.701Z   File "/usr/local/lib/python3.13/site-packages/apify/storages/_request_list.py", line 13, in <module>
2025-03-08T12:46:23.703Z     from crawlee.http_clients import HttpClient, HttpxHttpClient
2025-03-08T12:46:23.705Z   File "/usr/local/lib/python3.13/site-packages/crawlee/http_clients/__init__.py", line 6, in <module>
2025-03-08T12:46:23.707Z     from ._httpx import HttpxHttpClient
2025-03-08T12:46:23.710Z   File "/usr/local/lib/python3.13/site-packages/crawlee/http_clients/_httpx.py", line 13, in <module>
2025-03-08T12:46:23.712Z     from crawlee.fingerprint_suite import HeaderGenerator
2025-03-08T12:46:23.714Z   File "/usr/local/lib/python3.13/site-packages/crawlee/fingerprint_suite/__init__.py", line 1, in <module>
2025-03-08T12:46:23.717Z     from ._browserforge_adapter import BrowserforgeFingerprintGenerator as DefaultFingerprintGenerator
2025-03-08T12:46:23.732Z   File "/usr/local/lib/python3.13/site-packages/crawlee/fingerprint_suite/_browserforge_adapter.py", line 10, in <module>
2025-03-08T12:46:23.734Z     from browserforge.fingerprints import Fingerprint as bf_Fingerprint
2025-03-08T12:46:23.736Z   File "/usr/local/lib/python3.13/site-packages/browserforge/fingerprints/__init__.py", line 5, in <module>
2025-03-08T12:46:23.738Z     from browserforge.headers import Browser
2025-03-08T12:46:23.741Z   File "/usr/local/lib/python3.13/site-packages/browserforge/headers/__init__.py", line 5, in <module>
2025-03-08T12:46:23.743Z     from .generator import Browser, HeaderGenerator
2025-03-08T12:46:23.745Z   File "/usr/local/lib/python3.13/site-packages/browserforge/headers/generator.py", line 80, in <module>
2025-03-08T12:46:23.747Z     class HeaderGenerator:
2025-03-08T12:46:23.750Z     ...<470 lines>...
2025-03-08T12:46:23.752Z             )
2025-03-08T12:46:23.756Z   File "/usr/local/lib/python3.13/site-packages/browserforge/headers/generator.py", line 86, in HeaderGenerator
2025-03-08T12:46:23.759Z     input_generator_network = BayesianNetwork(DATA_DIR / "input-network.zip")
2025-03-08T12:46:23.761Z   File "/usr/local/lib/python3.13/site-packages/browserforge/bayesian_network.py", line 103, in __init__
2025-03-08T12:46:23.763Z     network_definition = extract_json(path)
2025-03-08T12:46:23.766Z   File "/usr/local/lib/python3.13/site-packages/browserforge/bayesian_network.py", line 288, in extract_json
2025-03-08T12:46:23.768Z     with zipfile.ZipFile(path, 'r') as zf:
2025-03-08T12:46:23.770Z          ~~~~~~~~~~~~~~~^^^^^^^^^^^
2025-03-08T12:46:23.772Z   File "/usr/local/lib/python3.13/zipfile/__init__.py", line 1367, in __init__
2025-03-08T12:46:23.774Z     self.fp = io.open(file, filemode)
2025-03-08T12:46:23.776Z               ~~~~~~~^^^^^^^^^^^^^^^^
2025-03-08T12:46:23.779Z FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.13/site-packages/browserforge/headers/data/input-network.zip'

  5. Erol444 commented on Mar 8, 2025

    @Erol444
    Erol444
    on Mar 8, 2025

    @vdusek you should look into this asap - new py actors will by default use 2.4.0 apify, which throws the error above when starting the actor on apify platform. Thanks to this issue in actor's requirements.txt I pinned apify==2.3.0 which fixes the error above (I was messing with Dockerfile at first).

  6. Pijukatel commented on Mar 10, 2025

    @Pijukatel
    Pijukatel
    on Mar 10, 2025
    Contributor

    Proposed solution:

    • Create new PyPi package apify_fingerprint_datapoints: feat: Fingerprint datapoints python package fingerprint-suite#353
    • Change browserforge to not download any files, but explicitly import those files from newly created package apify_fingerprint_datapoints. PR to be created from this branch
       after package is published to Pypi
    • Update all Apify requirements constraints to depend on the latest browserforge version
  7. vdusek
    added
    bugSomething isn't working.
    on Mar 10, 2025
  8. Pijukatel
    added 2 commits that reference this issue on Mar 12, 2025

    fix: Temporary workaround for `browserforge` import time code executi…

    17d914f

    fix: Update to `browserforge` workaround (#1075)

    2378cf8

  9. Pijukatel commented on Mar 13, 2025

    @Pijukatel
    Pijukatel
    on Mar 13, 2025
    Contributor

    It should be fixed now for new templates that will use new base docker image. It is quite an uggly import-time-patch workaround because the PR in the upstream third party repo did not get merged yet.

  10. Pijukatel
    closed this as completedon Mar 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions