Celery 3.1 with Django, django-celery, RabbitMQ and MacPorts

Celery 3.1 was released earlier this week and with it comes Django support out of the box. This means Django users can now use the Celery API directly rather than depending on django-celery (the libary is still needed if you want to make use of the database result backend or the Django periodic task admin). This also means you can now use the celery command directly rather than going through manage.py.

In this post I’ll cover the changes necessary to migrate to 3.1 from a previous installation as well as provide updated instructions for getting started from scratch. First, the migration…

1) Remove django-celery integration (even if you are using django-celery for the database result backend or the periodic task admin, you still need to do this). Remove the following from your Django settings (also remove from wsgi.py if present there):

2) If you’re using the default RabbitMQ broker (and default username / password), you won’t need this in your settings either:

3) To still use the database result backend from django-celery, ensure the following is in your Django settings:

4) Create an instance of the celery library for Django to use.

This code performs a number of functions:

  • sets the default DJANGO_SETTINGS_MODULE for the celery command-line program
  • creates an instance of the celery library
  • adds the Django settings module as a configuration source for Celery (allows you to configure Celery directly from your Django settings)
  • autodiscovers tasks within your installed apps

In my application structure I’ve placed this in a file called celery.py:

5) If you want to make use of the shared_task decorator as demonstrated in the Django example application, add the following to your application’s __init__.py:

This decorator returns a proxy that always points to the currently active Celery instance, allowing you to define tasks within multiple applications and still use a single Celery instance.

That’s it for changes. Now the final piece is making use of new commands for running Celery. To run a worker, you previously used the following:

$ python manage.py celery worker --loglevel=info

This is replaced with:

$ celery -A app worker -l info

Similarly, running Celery beat:

$ python manage.py celery beat

Is now:

$ celery beat -A app

Note that each command requires the name of your Django application (the one that contains celery.py) to be passed via -A.

For a fresh install, here’s a quick rundown of the 6 steps it takes to get up and running…

1) Install RabbitMQ

2) OPTIONAL – configure a rabbitmq user

3) Install Celery (django-celery is only necessary if you want to use the database result backend or the Django periodic task admin)

4) Configure your Django application (only necessary if using django-celery, be sure to also add django-celery to your installed apps)

5) Create an instance of the celery library for Django to use

In my application structure I’ve placed this in a file called celery.py:

6) Sync your DB (only necessary if using django-celery)

You can now run Celery using the following commands:

$ celery -A app worker -l info

$ celery beat -A app

6 thoughts on “Celery 3.1 with Django, django-celery, RabbitMQ and MacPorts”

    1. I usually keep my tasks in a directory called ‘tasks’ underneath my application directory. If you have a look at the folder structure screenshot above, you’ll see the directory between storage and templates. Celery knows to look for tasks in this directory via the auto discover directive:

      app.autodiscover_tasks(settings.INSTALLED_APPS, related_name=’tasks’)

      The docs have a ton of info regarding actually writing tasks. Give them a look.


  1. Hey Chris,

    I am very confused with all the documents now…
    i have used django-celery, celery(3.1), and rabbitmq all is configured according to docs.

    but when i my code calls “task” worker throws “KeyError” like this

    ERROR/MainProcess] Received unregistered task of type ‘celeryapp.tasks.process_transcoding’.
    The message has been ignored and discarded.

    Did you remember to import the module containing this task?
    Or maybe you are using relative imports?
    Please see http://bit.ly/gLye1c for more information.

    The full contents of the message body was:
    {‘utc’: True, ‘chord’: None, ‘args’: [2188L, u’/opt/ossite/media/videos/Screenshot_from_2013-12-23_122741.png’, u’/opt/ossite/media/thumbnails/Screenshot_from_2013-12-23_122741.jpeg’, ‘thumbnail’], ‘retries’: 0, ‘expires’: None, ‘task’: ‘celeryapp.tasks.process_transcoding’, ‘callbacks’: None, ‘errbacks’: None, ‘timelimit’: (None, None), ‘taskset’: None, ‘kwargs’: {}, ‘eta’: None, ‘id’: ‘141c4d1e-3681-4cba-9976-dd951adcde6a’} (393b)
    Traceback (most recent call last):
    File “/usr/local/lib/python2.7/dist-packages/celery/worker/consumer.py”, line 460, in on_task_received
    strategies[name](message, body,
    KeyError: ‘celeryapp.tasks.process_transcoding’

    Please Help Buddy.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s