celery_beat relations not found by postgres using docker

Question

I have a Django app with Nginx, Gunicorn, PostgreSQL and Celery that I've been dockerizing. When trying to add celery_beat in my docker-compose.yml, I get a django.db.utils.ProgrammingError: relation "django_celery_beat_periodictask" does not exist even though the migrations have been ran successfully.

Locally, starting celery_beat works fine, but here it returns this error and I have no idea where it could be coming from. Postgres doesn't seem to find the relation django_celery_beat_periodictask but it should since the migrations were done. Is there something I misunderstood ?

Here is my docker-compose.yml:

version: '3.8'

services:
  rabbitmq3:
    container_name: rabbitmq
    image: rabbitmq:3-alpine
    ports:
      - 5672:5672

  postgres:
    container_name: postgres
    hostname: postgres
    image: postgres:latest
    env_file:
      - env
    environment:
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=db
    ports:
      - "5432:5432"
    restart: on-failure
    volumes:
      - postgresql-data:/var/lib/postgresql/data

  django_gunicorn:
    container_name: django_gunicorn
    volumes:
      - static:/app/static
      - media:/app/media
    env_file:
      - env
    build:
      context: .
    ports: 
      - "8000:8000"
    command: sh -c "python manage.py migrate && python manage.py collectstatic --no-input && gunicorn main.wsgi:application --bind 0.0.0.0:8000"
    depends_on:
      - postgres

  nginx:
    container_name: nginx
    build: ./nginx
    volumes:
      - .:/code
      - static:/static
    ports:
      - "80:80"
    depends_on:
      - django_gunicorn

  celery:
    container_name: celery
    volumes:
      - media:/app/media
    build:
      context: .
    command: celery -A main worker -P eventlet -c 100 -l INFO
    env_file:
      - env
    restart: always
    depends_on:
      - rabbitmq3
      - postgres
      - django_gunicorn

  celery_beat:
    container_name: celery_beat
    command: celery -A main beat --scheduler django_celery_beat.schedulers:DatabaseScheduler -l INFO
    build:
      context: .
    environment:
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=db
    restart: on-failure
    depends_on:
      - rabbitmq3
      - postgres
      - django_gunicorn

volumes:
  postgresql-data:
  static:
  media:

And the full traceback:

celery beat v5.2.7 (dawn-chorus) is starting.
__    -    ... __   -        _
LocalTime -> 2022-07-26 08:19:04
Configuration ->
    . broker -> amqp://guest:**@rabbitmq3:5672//
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> django_celery_beat.schedulers.DatabaseScheduler

    . logfile -> [stderr]@%INFO
    . maxinterval -> 5.00 seconds (5s)
[2022-07-26 08:19:04,280: INFO/MainProcess] beat: Starting...
[2022-07-26 08:19:04,293: CRITICAL/MainProcess] beat raised exception <class 'django.db.utils.ProgrammingError'>: ProgrammingError('relation "django_celery_beat_periodictask" does not exist\nLINE 1: ...ango_celery_beat_periodictask"."description" FROM "django_ce...\n                                                             ^\n')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "django_celery_beat_periodictask" does not exist
LINE 1: ...ango_celery_beat_periodictask"."description" FROM "django_ce...
                                                             ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/celery/apps/beat.py", line 105, in start_scheduler
    service.start()
  File "/usr/local/lib/python3.10/site-packages/celery/beat.py", line 636, in start
    humanize_seconds(self.scheduler.max_interval))
  File "/usr/local/lib/python3.10/site-packages/kombu/utils/objects.py", line 30, in __get__
    return super().__get__(instance, owner)
  File "/usr/local/lib/python3.10/functools.py", line 981, in __get__
    val = self.func(instance)
  File "/usr/local/lib/python3.10/site-packages/celery/beat.py", line 679, in scheduler
    return self.get_scheduler()
  File "/usr/local/lib/python3.10/site-packages/celery/beat.py", line 670, in get_scheduler
    return symbol_by_name(self.scheduler_cls, aliases=aliases)(
  File "/usr/local/lib/python3.10/site-packages/django_celery_beat/schedulers.py", line 228, in __init__
    Scheduler.__init__(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/celery/beat.py", line 271, in __init__
    self.setup_schedule()
  File "/usr/local/lib/python3.10/site-packages/django_celery_beat/schedulers.py", line 236, in setup_schedule
    self.install_default_entries(self.schedule)
  File "/usr/local/lib/python3.10/site-packages/django_celery_beat/schedulers.py", line 358, in schedule
    self._schedule = self.all_as_schedule()
  File "/usr/local/lib/python3.10/site-packages/django_celery_beat/schedulers.py", line 242, in all_as_schedule
    for model in self.Model.objects.enabled():
  File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 320, in __iter__
    self._fetch_all()
  File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 1507, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 57, in __iter__
    results = compiler.execute_sql(
  File "/usr/local/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1361, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "/usr/local/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "django_celery_beat_periodictask" does not exist
LINE 1: ...ango_celery_beat_periodictask"."description" FROM "django_ce...
                                                             ^

[2022-07-26 08:19:04,296: WARNING/MainProcess] Traceback (most recent call last):
[2022-07-26 08:19:04,296: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 89, in _execute
[2022-07-26 08:19:04,296: WARNING/MainProcess]
[2022-07-26 08:19:04,296: WARNING/MainProcess] return self.cursor.execute(sql, params)
[2022-07-26 08:19:04,297: WARNING/MainProcess] psycopg2.errors
[2022-07-26 08:19:04,297: WARNING/MainProcess] .
[2022-07-26 08:19:04,297: WARNING/MainProcess] UndefinedTable
[2022-07-26 08:19:04,297: WARNING/MainProcess] :
[2022-07-26 08:19:04,297: WARNING/MainProcess] relation "django_celery_beat_periodictask" does not exist
LINE 1: ...ango_celery_beat_periodictask"."description" FROM "django_ce...
                                                             ^
[2022-07-26 08:19:04,297: WARNING/MainProcess]
The above exception was the direct cause of the following exception:
[2022-07-26 08:19:04,297: WARNING/MainProcess] Traceback (most recent call last):
[2022-07-26 08:19:04,297: WARNING/MainProcess]   File "/usr/local/bin/celery", line 8, in <module>
[2022-07-26 08:19:04,298: WARNING/MainProcess]
[2022-07-26 08:19:04,298: WARNING/MainProcess] sys.exit(main())
[2022-07-26 08:19:04,298: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/celery/__main__.py", line 15, in main
[2022-07-26 08:19:04,298: WARNING/MainProcess]
[2022-07-26 08:19:04,298: WARNING/MainProcess] sys.exit(_main())
[2022-07-26 08:19:04,298: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/celery/bin/celery.py", line 217, in main
[2022-07-26 08:19:04,299: WARNING/MainProcess]
[2022-07-26 08:19:04,299: WARNING/MainProcess] return celery(auto_envvar_prefix="CELERY")
[2022-07-26 08:19:04,299: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
[2022-07-26 08:19:04,299: WARNING/MainProcess]
[2022-07-26 08:19:04,299: WARNING/MainProcess] return self.main(*args, **kwargs)
[2022-07-26 08:19:04,300: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
[2022-07-26 08:19:04,300: WARNING/MainProcess]
[2022-07-26 08:19:04,300: WARNING/MainProcess] rv = self.invoke(ctx)
[2022-07-26 08:19:04,300: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
[2022-07-26 08:19:04,301: WARNING/MainProcess]
[2022-07-26 08:19:04,301: WARNING/MainProcess] return _process_result(sub_ctx.command.invoke(sub_ctx))
[2022-07-26 08:19:04,301: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
[2022-07-26 08:19:04,301: WARNING/MainProcess]
[2022-07-26 08:19:04,302: WARNING/MainProcess] return ctx.invoke(self.callback, **ctx.params)
[2022-07-26 08:19:04,302: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
[2022-07-26 08:19:04,302: WARNING/MainProcess]
[2022-07-26 08:19:04,302: WARNING/MainProcess] return __callback(*args, **kwargs)
[2022-07-26 08:19:04,302: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
[2022-07-26 08:19:04,302: WARNING/MainProcess]
[2022-07-26 08:19:04,302: WARNING/MainProcess] return f(get_current_context(), *args, **kwargs)
[2022-07-26 08:19:04,303: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/celery/bin/base.py", line 134, in caller
[2022-07-26 08:19:04,303: WARNING/MainProcess]
[2022-07-26 08:19:04,303: WARNING/MainProcess] return f(ctx, *args, **kwargs)
[2022-07-26 08:19:04,303: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/celery/bin/beat.py", line 72, in beat
[2022-07-26 08:19:04,304: WARNING/MainProcess]
[2022-07-26 08:19:04,304: WARNING/MainProcess] return beat().run()
[2022-07-26 08:19:04,304: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/celery/apps/beat.py", line 77, in run
[2022-07-26 08:19:04,304: WARNING/MainProcess]
[2022-07-26 08:19:04,304: WARNING/MainProcess] self.start_scheduler()
[2022-07-26 08:19:04,304: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/celery/apps/beat.py", line 105, in start_scheduler
[2022-07-26 08:19:04,305: WARNING/MainProcess]
[2022-07-26 08:19:04,305: WARNING/MainProcess] service.start()
[2022-07-26 08:19:04,305: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/celery/beat.py", line 636, in start
[2022-07-26 08:19:04,305: WARNING/MainProcess]
[2022-07-26 08:19:04,305: WARNING/MainProcess] humanize_seconds(self.scheduler.max_interval))
[2022-07-26 08:19:04,305: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/kombu/utils/objects.py", line 30, in __get__
[2022-07-26 08:19:04,305: WARNING/MainProcess]
[2022-07-26 08:19:04,306: WARNING/MainProcess] return super().__get__(instance, owner)
[2022-07-26 08:19:04,306: WARNING/MainProcess]   File "/usr/local/lib/python3.10/functools.py", line 981, in __get__
[2022-07-26 08:19:04,307: WARNING/MainProcess]
[2022-07-26 08:19:04,307: WARNING/MainProcess] val = self.func(instance)
[2022-07-26 08:19:04,307: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/celery/beat.py", line 679, in scheduler
[2022-07-26 08:19:04,307: WARNING/MainProcess]
[2022-07-26 08:19:04,307: WARNING/MainProcess] return self.get_scheduler()
[2022-07-26 08:19:04,307: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/celery/beat.py", line 670, in get_scheduler
[2022-07-26 08:19:04,308: WARNING/MainProcess]
[2022-07-26 08:19:04,308: WARNING/MainProcess] return symbol_by_name(self.scheduler_cls, aliases=aliases)(
[2022-07-26 08:19:04,308: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/django_celery_beat/schedulers.py", line 228, in __init__
[2022-07-26 08:19:04,308: WARNING/MainProcess]
[2022-07-26 08:19:04,308: WARNING/MainProcess] Scheduler.__init__(self, *args, **kwargs)
[2022-07-26 08:19:04,308: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/celery/beat.py", line 271, in __init__
[2022-07-26 08:19:04,309: WARNING/MainProcess]
[2022-07-26 08:19:04,309: WARNING/MainProcess] self.setup_schedule()
[2022-07-26 08:19:04,309: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/django_celery_beat/schedulers.py", line 236, in setup_schedule
[2022-07-26 08:19:04,309: WARNING/MainProcess]
[2022-07-26 08:19:04,309: WARNING/MainProcess] self.install_default_entries(self.schedule)
[2022-07-26 08:19:04,309: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/django_celery_beat/schedulers.py", line 358, in schedule
[2022-07-26 08:19:04,309: WARNING/MainProcess]
[2022-07-26 08:19:04,310: WARNING/MainProcess] self._schedule = self.all_as_schedule()
[2022-07-26 08:19:04,310: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/django_celery_beat/schedulers.py", line 242, in all_as_schedule
[2022-07-26 08:19:04,310: WARNING/MainProcess]
[2022-07-26 08:19:04,310: WARNING/MainProcess] for model in self.Model.objects.enabled():
[2022-07-26 08:19:04,310: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 320, in __iter__
[2022-07-26 08:19:04,310: WARNING/MainProcess]
[2022-07-26 08:19:04,310: WARNING/MainProcess] self._fetch_all()
[2022-07-26 08:19:04,310: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 1507, in _fetch_all
[2022-07-26 08:19:04,311: WARNING/MainProcess]
[2022-07-26 08:19:04,311: WARNING/MainProcess] self._result_cache = list(self._iterable_class(self))
[2022-07-26 08:19:04,311: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 57, in __iter__
[2022-07-26 08:19:04,311: WARNING/MainProcess]
[2022-07-26 08:19:04,311: WARNING/MainProcess] results = compiler.execute_sql(
[2022-07-26 08:19:04,311: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1361, in execute_sql
[2022-07-26 08:19:04,312: WARNING/MainProcess]
[2022-07-26 08:19:04,312: WARNING/MainProcess] cursor.execute(sql, params)
[2022-07-26 08:19:04,312: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 67, in execute
[2022-07-26 08:19:04,312: WARNING/MainProcess]
[2022-07-26 08:19:04,312: WARNING/MainProcess] return self._execute_with_wrappers(
[2022-07-26 08:19:04,312: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
[2022-07-26 08:19:04,313: WARNING/MainProcess]
[2022-07-26 08:19:04,313: WARNING/MainProcess] return executor(sql, params, many, context)
[2022-07-26 08:19:04,313: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
[2022-07-26 08:19:04,313: WARNING/MainProcess]
[2022-07-26 08:19:04,313: WARNING/MainProcess] with self.db.wrap_database_errors:
[2022-07-26 08:19:04,313: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
[2022-07-26 08:19:04,313: WARNING/MainProcess]
[2022-07-26 08:19:04,313: WARNING/MainProcess] raise dj_exc_value.with_traceback(traceback) from exc_value
[2022-07-26 08:19:04,314: WARNING/MainProcess]   File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 89, in _execute
[2022-07-26 08:19:04,314: WARNING/MainProcess]
[2022-07-26 08:19:04,314: WARNING/MainProcess] return self.cursor.execute(sql, params)
[2022-07-26 08:19:04,314: WARNING/MainProcess] django.db.utils
[2022-07-26 08:19:04,314: WARNING/MainProcess] .
[2022-07-26 08:19:04,314: WARNING/MainProcess] ProgrammingError
[2022-07-26 08:19:04,314: WARNING/MainProcess] :
[2022-07-26 08:19:04,315: WARNING/MainProcess] relation "django_celery_beat_periodictask" does not exist
LINE 1: ...ango_celery_beat_periodictask"."description" FROM "django_ce...

And postgres returns this error:

postgres         | 2022-07-26 08:48:01.489 UTC [71] ERROR:  relation "django_celery_beat_periodictask" does not exist at character 1078
postgres         | 2022-07-26 08:48:01.489 UTC [71] STATEMENT:  SELECT "django_celery_beat_periodictask"."id", "django_celery_beat_periodictask"."name", "django_celery_beat_periodictask"."task", "django_celery_beat_periodictask"."interval_id", "django_celery_beat_periodictask"."crontab_id", "django_celery_beat_periodictask"."solar_id", "django_celery_beat_periodictask"."clocked_id", "django_celery_beat_periodictask"."args", "django_celery_beat_periodictask"."kwargs", "django_celery_beat_periodictask"."queue", "django_celery_beat_periodictask"."exchange", "django_celery_beat_periodictask"."routing_key", "django_celery_beat_periodictask"."headers", "django_celery_beat_periodictask"."priority", "django_celery_beat_periodictask"."expires", "django_celery_beat_periodictask"."expire_seconds", "django_celery_beat_periodictask"."one_off", "django_celery_beat_periodictask"."start_time", "django_celery_beat_periodictask"."enabled", "django_celery_beat_periodictask"."last_run_at", "django_celery_beat_periodictask"."total_run_count", "django_celery_beat_periodictask"."date_changed", "django_celery_beat_periodictask"."description" 
FROM "django_celery_beat_periodictask" WHERE "django_celery_beat_periodictask"."enabled"

EDIT

I've tried running migrations inside of my celery_beat container and it successfully started. However it doesn't seem to be linked to Celery nor Django as setting a periodic task on the admin panel doesn't change anything and no task is sent to Celery.

So I think the issue lays in the celery_beat not being able to access the migrations...somewhere

EDIT 2

If I try to set the celery_beat command to celery -A main beat then it works perfectly. The problem probably comes from the definition of the database scheduler but I have no clue as to what I should change it into.

Answer 1

This happens if: you have any external module (django_celery_beat in this case) which is installed (in venv) and added to INSTALLED_APPS, and the associated migration of the respective module is not done.

You can manually migrate it with with you env enabled:

python manage.py migrate django_celery_beat

It'll create the associated table in the configured database and respective entries in the django_migrations table.

Once the associated tables are in place, it'll no longer raise exceptions.