AutoDL not working with MinIO backend (migrating from Google Cloud) {google-cloud-merged branch} #3548
Comments
|
Hi @ObadaS, can you put also the command you are trying to run? And a quick summary of what we are trying to achieve here? Thank you. |
|
@Didayolo I updated the original post with more details |
|
I made a new branch to try things out. The docker images successfully builds, but the Django and Worker_Site containers crashes on startup with the following errors: Django Traceback (most recent call last):
File "scripts/initialize.py", line 15, in <module>
from configurations import importer
File "/usr/local/lib/python2.7/site-packages/configurations/__init__.py", line 2, in <module>
from .base import Settings, Configuration
File "/usr/local/lib/python2.7/site-packages/configurations/base.py", line 7, in <module>
from .utils import uppercase_attributes
File "/usr/local/lib/python2.7/site-packages/configurations/utils.py", line 5, in <module>
from django.utils.importlib import import_module
ImportError: No module named importlib
Traceback (most recent call last):
File "scripts/initialize.py", line 15, in <module>
from configurations import importer
File "/usr/local/lib/python2.7/site-packages/configurations/__init__.py", line 2, in <module>
from .base import Settings, Configuration
File "/usr/local/lib/python2.7/site-packages/configurations/base.py", line 7, in <module>
from .utils import uppercase_attributes
File "/usr/local/lib/python2.7/site-packages/configurations/utils.py", line 5, in <module>
from django.utils.importlib import import_module
ImportError: No module named importlib
/app/docker/run_django.sh: line 31: gunicorn: command not foundWorker_Site Traceback (most recent call last):
File "/usr/local/bin/celery", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/site-packages/celery/__main__.py", line 30, in main
main()
File "/usr/local/lib/python2.7/site-packages/celery/bin/celery.py", line 81, in main
cmd.execute_from_commandline(argv)
File "/usr/local/lib/python2.7/site-packages/celery/bin/celery.py", line 793, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/usr/local/lib/python2.7/site-packages/celery/bin/base.py", line 309, in execute_from_commandline
argv = self.setup_app_from_commandline(argv)
File "/usr/local/lib/python2.7/site-packages/celery/bin/base.py", line 469, in setup_app_from_commandline
self.app = self.find_app(app)
File "/usr/local/lib/python2.7/site-packages/celery/bin/base.py", line 489, in find_app
return find_app(app, symbol_by_name=self.symbol_by_name)
File "/usr/local/lib/python2.7/site-packages/celery/app/utils.py", line 235, in find_app
sym = symbol_by_name(app, imp=imp)
File "/usr/local/lib/python2.7/site-packages/celery/bin/base.py", line 492, in symbol_by_name
return symbol_by_name(name, imp=imp)
File "/usr/local/lib/python2.7/site-packages/kombu/utils/__init__.py", line 96, in symbol_by_name
module = imp(module_name, package=package, **kwargs)
File "/usr/local/lib/python2.7/site-packages/celery/utils/imports.py", line 101, in import_from_cwd
return imp(module, package=package)
File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/app/codalab/codalab/__init__.py", line 5, in <module>
from .celery import app as celery_app
File "/app/codalab/codalab/celery.py", line 12, in <module>
from configurations import importer
File "/usr/local/lib/python2.7/site-packages/configurations/__init__.py", line 2, in <module>
from .base import Settings, Configuration
File "/usr/local/lib/python2.7/site-packages/configurations/base.py", line 7, in <module>
from .utils import uppercase_attributes
File "/usr/local/lib/python2.7/site-packages/configurations/utils.py", line 5, in <module>
from django.utils.importlib import import_module
ImportError: No module named importlib |
|
I made some other changes : a11a065 The problem is present in the I have some new problems now: Traceback (most recent call last):
File "scripts/initialize.py", line 231, in <module>
django.setup()
File "/usr/local/lib/python2.7/site-packages/django/__init__.py", line 27, in setup
apps.populate(settings.INSTALLED_APPS)
File "/usr/local/lib/python2.7/site-packages/django/apps/registry.py", line 85, in populate
app_config = AppConfig.create(entry)
File "/usr/local/lib/python2.7/site-packages/django/apps/config.py", line 94, in create
module = import_module(entry)
File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
ImportError: No module named django_celery_results |
|
Some more progress on this: 4535b4e I think the next problem comes from the migration that gets applied on boot for the django container. The problem now is that the django container exits (code 3) for seemingly no reason: django | WARNINGS:
django | ?: (urls.W005) URL namespace 'admin' isn't unique. You may not be able to reverse all URLs in this namespace
django | Operations to perform:
django | Apply all migrations: account, admin, auth, authenz, captcha, contenttypes, coopetitions, customizer, django_celery_results, forums, guardian, health, jobs, oauth2_provider, queues, sessions, sites, socialaccount, teams, web
django | Running migrations:
django | No migrations to apply.
django | Initial data has been detected in the database: skipping all inserts. Running data migration...
django | Data migration complete.
django exited with code 3 |
|
I fixed the problem reported in the last comment by adding yet another missing package : fa20ada But now, it seems like we have another problem, maybe related to migrations and the database : 2025-01-30 10:27:24,291 ERROR Internal Server Error: /
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner
response = get_response(request)
File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 244, in _legacy_get_response
response = middleware_method(request)
File "/app/codalab/apps/web/middleware.py", line 118, in process_request
config, created = Configuration.objects.get_or_create(pk=1)
File "/usr/local/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 464, in get_or_create
return self.get(**lookup), False
File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 374, in get
num = len(clone)
File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 232, in __len__
self._fetch_all()
File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 1121, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 53, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 899, in execute_sql
raise original_exception
ProgrammingError: column customizer_configuration.front_page_message does not exist
LINE 1: ..."."id", "customizer_configuration"."header_logo", "customize...
^All migrations ran successfully before the error appeared, so either the branch is missing some important migration, or the problem comes from somewhere else. Also, for some reason, |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When changing the
.envfile to use MinIO instead of Google Cloud on an existing deployment, then trying to download a file that's stored in MinIO (the files were copied from the Google cloud buckets into the MinIO buckets), we get this error.I am pretty sure that MinIO support was added later, starting (maybe) with this commit which is not present in the current branch for autodl.
The text was updated successfully, but these errors were encountered: