Django management command won't work in cron

Question

I am having problems scheduling a manage.py celery call myapp.tasks.mytask with my user crontab, in that when cron tries to run the job, it gets this in stderr (which gets mailed to me, as /var/mail/kal)

Unknown command: 'celery'
Type 'manage.py help' for usage.

The same command works completely from a regular bash login shell, but it won't work in crontab.

I am doing this on Debian wheezy:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 7.0 (wheezy)
Release:        7.0
Codename:       wheezy

I have read many similar questions on StackOverflow and tried many of the suggested solutions. None of them have worked for me so far. Here are the solutions I have tried so far:

First, I made sure to specify relevant environment variables in the crontab:

SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

NOTE: these stay in place in all of the following solutions.

1. Using full paths to the python executable and manage.py scripts

* * * * * /home/kal/.virtualenvs/foo_dev/bin/python /home/kal/foo/manage.py celery call myapp.tasks.mytask

2. cd'ing into the project path first

* * * * * cd /home/kal/foo && /home/kal/.virtualenvs/foo_dev/bin/python ./manage.py celery call myapp.tasks.mytask

3. Wrapping everything in a bash script

Content of ~/mytask.sh:

#!/usr/bin/env bash
source /home/kal/.virtualenvs/foo_dev/bin/activate;
cd /home/kal/foo;
./manage.py celery call myapp.tasks.mytask;

The crontab line:

* * * * * ~/mytask.sh

I even modified myproj/settings.py to output sys.path and sys.executable to stderr and compared the output between cron and the login shell, and they are exactly the same:

Output from cron job:

sys.executable:
    /home/kal/.virtualenvs/foo_dev/bin/python

Content of sys.path:
    /home/kal/foo
    /home/kal/.virtualenvs/foo_dev/src/bootstrap
    /home/kal/.virtualenvs/foo_dev/src/django-json-rpc
    /home/kal/.virtualenvs/foo_dev/lib/python2.7
    /home/kal/.virtualenvs/foo_dev/lib/python2.7/plat-linux2
    /home/kal/.virtualenvs/foo_dev/lib/python2.7/lib-tk
    /home/kal/.virtualenvs/foo_dev/lib/python2.7/lib-old
    /home/kal/.virtualenvs/foo_dev/lib/python2.7/lib-dynload
    /usr/lib/python2.7
    /usr/lib/python2.7/plat-linux2
    /usr/lib/python2.7/lib-tk
    /home/kal/.virtualenvs/foo_dev/local/lib/python2.7/site-packages
    /home/kal/foo

Output from Bash login shell:

sys.executable:
    /home/kal/.virtualenvs/foo_dev/bin/python

Content of sys.path:
    /home/kal/foo
    /home/kal/.virtualenvs/foo_dev/src/bootstrap
    /home/kal/.virtualenvs/foo_dev/src/django-json-rpc
    /home/kal/.virtualenvs/foo_dev/lib/python2.7
    /home/kal/.virtualenvs/foo_dev/lib/python2.7/plat-linux2
    /home/kal/.virtualenvs/foo_dev/lib/python2.7/lib-tk
    /home/kal/.virtualenvs/foo_dev/lib/python2.7/lib-old
    /home/kal/.virtualenvs/foo_dev/lib/python2.7/lib-dynload
    /usr/lib/python2.7
    /usr/lib/python2.7/plat-linux2
    /usr/lib/python2.7/lib-tk
    /home/kal/.virtualenvs/foo_dev/local/lib/python2.7/site-packages
    /home/kal/foo

I am completely baffled.

Answer 1

I found the cause of the problem.

It is very very subtle.

The problem is two fold:

1. There is no USER environment variable in a cron job; only LOGNAME;
2. When manage.py is run with a management command specified, Django quietly fails over to blank settings if an exception is raised during the import of the settings module.

My settings module was trying to reference os.environ['USER'], which doesn't exist in cron's environment. So importing the settings module causes an exception to be raised, and Django quietly fails over to blank settings, which means blank INSTALLED_APPS and no celery command!

Answer 2

Forget cron. Use the celerybeat_scheduler.