Can't get the project to work

[jakub-bochenski]

System configuration

if [ ! -d "tools/scancode.io" ]; then
	git clone https://github.com/aboutcode-org/scancode.io.git tools/scancode.io
fi
cd tools/scancode.io
make envfile
docker compose build
docker compose up -d --wait

tools/scancode.io$ docker compose ps
NAME              IMAGE                           COMMAND                  SERVICE   CREATED         STATUS         PORTS
scancodeio-db-1   docker.io/library/postgres:13   "docker-entrypoint.s…"   db        3 minutes ago   Up 3 minutes   5432/tcp

To Reproduce
cd tools/scancode.io && PROJECT_NAME=scancode-$(date +%s) && docker compose run --rm web scanpipe create-project $PROJECT_NAME --input-url docker://library/ghost:6.0.5 --pipeline analyze_docker_image --execute && docker compose run --rm web scanpipe output --project $PROJECT_NAME --format json --print:

Traceback (most recent call last):
  File "/opt/scancodeio/.venv/lib/python3.13/site-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/opt/scancodeio/.venv/lib/python3.13/site-packages/psycopg/cursor.py", line 97, in execute
    raise ex.with_traceback(None)
psycopg.errors.UndefinedTable: relation "scanpipe_project" does not exist
LINE 1: SELECT 1 AS "a" FROM "scanpipe_project" WHERE "scanpipe_proj...
                             ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/scancodeio/.venv/bin/scanpipe", line 7, in <module>
    sys.exit(command_line())
             ~~~~~~~~~~~~^^
  File "/opt/scancodeio/scancodeio/__init__.py", line 98, in command_line
    execute_from_command_line(sys.argv)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/opt/scancodeio/.venv/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
    ~~~~~~~~~~~~~~~^^
  File "/opt/scancodeio/.venv/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/opt/scancodeio/.venv/lib/python3.13/site-packages/django/core/management/base.py", line 413, in run_from_argv
    self.execute(*args, **cmd_options)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/scancodeio/.venv/lib/python3.13/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
  File "/opt/scancodeio/scanpipe/management/commands/create-project.py", line 42, in handle
    self.create_project(
    ~~~~~~~~~~~~~~~~~~~^
        name=options["name"],
        ^^^^^^^^^^^^^^^^^^^^^
    ...<8 lines>...
        create_global_webhook=not options["no_global_webhook"],
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/scancodeio/scanpipe/management/commands/__init__.py", line 554, in create_project
    return create_project(
        name=name,
    ...<9 lines>...
        command=self,
    )
  File "/opt/scancodeio/scanpipe/management/commands/__init__.py", line 457, in create_project
    project.full_clean(exclude=["slug"])
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/opt/scancodeio/.venv/lib/python3.13/site-packages/django/db/models/base.py", line 1636, in full_clean
    self.validate_unique(exclude=exclude)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/opt/scancodeio/.venv/lib/python3.13/site-packages/django/db/models/base.py", line 1378, in validate_unique
    errors = self._perform_unique_checks(unique_checks)
  File "/opt/scancodeio/.venv/lib/python3.13/site-packages/django/db/models/base.py", line 1488, in _perform_unique_checks
    if qs.exists():
       ~~~~~~~~~^^
  File "/opt/scancodeio/.venv/lib/python3.13/site-packages/django/db/models/query.py", line 1288, in exists
    return self.query.has_results(using=self.db)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/opt/scancodeio/.venv/lib/python3.13/site-packages/django/db/models/sql/query.py", line 660, in has_results
    return compiler.has_results()
           ~~~~~~~~~~~~~~~~~~~~^^
  File "/opt/scancodeio/.venv/lib/python3.13/site-packages/django/db/models/sql/compiler.py", line 1542, in has_results
    return bool(self.execute_sql(SINGLE))
                ~~~~~~~~~~~~~~~~^^^^^^^^
  File "/opt/scancodeio/.venv/lib/python3.13/site-packages/django/db/models/sql/compiler.py", line 1574, in execute_sql
    cursor.execute(sql, params)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/opt/scancodeio/.venv/lib/python3.13/site-packages/django/db/backends/utils.py", line 79, in execute
    return self._execute_with_wrappers(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        sql, params, many=False, executor=self._execute
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/scancodeio/.venv/lib/python3.13/site-packages/django/db/backends/utils.py", line 92, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/scancodeio/.venv/lib/python3.13/site-packages/django/db/backends/utils.py", line 100, in _execute
    with self.db.wrap_database_errors:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/scancodeio/.venv/lib/python3.13/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/scancodeio/.venv/lib/python3.13/site-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/opt/scancodeio/.venv/lib/python3.13/site-packages/psycopg/cursor.py", line 97, in execute
    raise ex.with_traceback(None)
django.db.utils.ProgrammingError: relation "scanpipe_project" does not exist
LINE 1: SELECT 1 AS "a" FROM "scanpipe_project" WHERE "scanpipe_proj...
                             ^

Expected behavior
It works

[jakub-bochenski]

Some thing if I use the v35.3.0

[jakub-bochenski]

I think docker compose up failed because a port was already taken.
Why does it run a web server on :80 if I want to use CLI?

Anyhow, after correcting this:

$ docker compose -f tools/scancode.io/docker-compose.yml ps
NAME                  IMAGE                            COMMAND                  SERVICE   CREATED          STATUS                    PORTS
scancodeio-clamav-1   docker.io/clamav/clamav:latest   "/init"                  clamav    10 minutes ago   Up 10 minutes (healthy)   3310/tcp, 7357/tcp
scancodeio-db-1       docker.io/library/postgres:13    "docker-entrypoint.s…"   db        10 minutes ago   Up 10 minutes             5432/tcp
scancodeio-nginx-1    docker.io/library/nginx:alpine   "/docker-entrypoint.…"   nginx     10 minutes ago   Up About a minute         0.0.0.0:80->80/tcp, [::]:80->80/tcp, 0.0.0.0:443->443/tcp, [::]:443->443/tcp
scancodeio-redis-1    docker.io/library/redis:latest   "docker-entrypoint.s…"   redis     10 minutes ago   Up 10 minutes             6379/tcp
scancodeio-web-1      scancodeio-web                   "wait-for-it --stric…"   web       10 minutes ago   Up 10 minutes             8000/tcp
scancodeio-worker-1   scancodeio-worker                "wait-for-it --stric…"   worker    10 minutes ago   Up 10 minutes             

I'm still getting errors:

 cd tools/scancode.io && PROJECT_NAME=scancode-$(date +%s) && docker compose run --rm web scanpipe create-project $PROJECT_NAME --input-url docker://library/ghost:6.0.5 --pipeline analyze_docker_image --execute && docker compose run --rm web scanpipe output --project $PROJECT_NAME --format json --print
 Container scancodeio-db-1  Running
Traceback (most recent call last):
  File "/usr/local/lib/python3.13/pathlib/_local.py", line 722, in mkdir
    os.mkdir(self, mode)
    ~~~~~~~~^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/var/scancodeio/workspace/projects/scancode-1756991314-a788d3ed/input'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.13/pathlib/_local.py", line 722, in mkdir
    os.mkdir(self, mode)
    ~~~~~~~~^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/var/scancodeio/workspace/projects/scancode-1756991314-a788d3ed'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/scancodeio/.venv/bin/scanpipe", line 7, in <module>
    sys.exit(command_line())
             ~~~~~~~~~~~~^^
  File "/opt/scancodeio/scancodeio/__init__.py", line 98, in command_line
    execute_from_command_line(sys.argv)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/opt/scancodeio/.venv/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
    ~~~~~~~~~~~~~~~^^
  File "/opt/scancodeio/.venv/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/opt/scancodeio/.venv/lib/python3.13/site-packages/django/core/management/base.py", line 413, in run_from_argv
    self.execute(*args, **cmd_options)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/scancodeio/.venv/lib/python3.13/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
  File "/opt/scancodeio/scanpipe/management/commands/create-project.py", line 42, in handle
    self.create_project(
    ~~~~~~~~~~~~~~~~~~~^
        name=options["name"],
        ^^^^^^^^^^^^^^^^^^^^^
    ...<8 lines>...
        create_global_webhook=not options["no_global_webhook"],
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/scancodeio/scanpipe/management/commands/__init__.py", line 554, in create_project
    return create_project(
        name=name,
    ...<9 lines>...
        command=self,
    )
  File "/opt/scancodeio/scanpipe/management/commands/__init__.py", line 470, in create_project
    project.save(**save_kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/opt/scancodeio/scanpipe/models.py", line 632, in save
    self.setup_work_directory()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/opt/scancodeio/scanpipe/models.py", line 827, in setup_work_directory
    Path(self.work_directory, subdirectory).mkdir(parents=True, exist_ok=True)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/pathlib/_local.py", line 726, in mkdir
    self.parent.mkdir(parents=True, exist_ok=True)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/pathlib/_local.py", line 726, in mkdir
    self.parent.mkdir(parents=True, exist_ok=True)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/pathlib/_local.py", line 722, in mkdir
    os.mkdir(self, mode)
    ~~~~~~~~^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/var/scancodeio/workspace/projects'

[tdruez]

Why does it run a web server on :80 if I want to use CLI?

The docker compose file is provided to run the full web server stack.

If you only want to run the CLI, you can try the following as a start: https://scancodeio.readthedocs.io/en/latest/quickstart.html#run-a-scan-no-installation-required

Based on your provided data, this should be something like:

docker run --rm ghcr.io/aboutcode-org/scancode.io:latest \
  run analyze_docker_image docker://library/ghost:6.0.5 \
  > results.json

See also https://scancodeio.readthedocs.io/en/latest/command-line-interface.html#run-pipeline-name-pipeline-name-input-location

[jakub-bochenski]

I was following the instruction at https://scancodeio.readthedocs.io/en/latest/tutorial_cli_analyze_docker_image.html
they tell you to use docker compose -f ${PWD}/docker-compose.yml. You might want to update the documentation?