Tag: Celery

Celery 3.1 with Django, django-celery, RabbitMQ and MacPorts

Celery 3.1 was released earlier this week and with it comes Django support out of the box. This means Django users can now use the Celery API directly rather than depending on django-celery (the libary is still needed if you want to make use of the database result backend or the Django periodic task admin). This also means you can now use the celery command directly rather than going through manage.py.

In this post I’ll cover the changes necessary to migrate to 3.1 from a previous installation as well as provide updated instructions for getting started from scratch. First, the migration…

1) Remove django-celery integration (even if you are using django-celery for the database result backend or the periodic task admin, you still need to do this). Remove the following from your Django settings (also remove from wsgi.py if present there):


import djcelery
djcelery.setup_loader()

view raw

settings.py

hosted with ❤ by GitHub

2) If you’re using the default RabbitMQ broker (and default username / password), you won’t need this in your settings either:


BROKER_URL = 'amqp://guest:guest@localhost:5672//'

view raw

settings.py

hosted with ❤ by GitHub

3) To still use the database result backend from django-celery, ensure the following is in your Django settings:


CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'

view raw

settings.py

hosted with ❤ by GitHub

4) Create an instance of the celery library for Django to use.


from __future__ import absolute_import
import os
from celery import Celery
from django.conf import settings
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'conf.settings')
app = Celery('app.celery')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(settings.INSTALLED_APPS, related_name='tasks')

view raw

celery.py

hosted with ❤ by GitHub

This code performs a number of functions:

  • sets the default DJANGO_SETTINGS_MODULE for the celery command-line program
  • creates an instance of the celery library
  • adds the Django settings module as a configuration source for Celery (allows you to configure Celery directly from your Django settings)
  • autodiscovers tasks within your installed apps

In my application structure I’ve placed this in a file called celery.py:

5) If you want to make use of the shared_task decorator as demonstrated in the Django example application, add the following to your application’s __init__.py:


from __future__ import absolute_import
# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app

view raw

__init__.py

hosted with ❤ by GitHub

This decorator returns a proxy that always points to the currently active Celery instance, allowing you to define tasks within multiple applications and still use a single Celery instance.

That’s it for changes. Now the final piece is making use of new commands for running Celery. To run a worker, you previously used the following:

$ python manage.py celery worker --loglevel=info

This is replaced with:

$ celery -A app worker -l info

Similarly, running Celery beat:

$ python manage.py celery beat

Is now:

$ celery beat -A app

Note that each command requires the name of your Django application (the one that contains celery.py) to be passed via -A.

For a fresh install, here’s a quick rundown of the 6 steps it takes to get up and running…

1) Install RabbitMQ


# required ports
sudo port install erlang +ssl
sudo port install rabbitmq-server
# start the server on system boot
sudo port load rabbitmq-server
# OPTIONAL – install the librabbitmq C library (it's faster thant the default amqp)
sudo pip install librabbitmq

view raw

installs.sh

hosted with ❤ by GitHub

2) OPTIONAL – configure a rabbitmq user


# RabbitMQ installs with a default username / password of guest / guest
# you can change that by creating a new user
rabbitmqctl add_user myuser mypassword
rabbitmqctl add_vhost myvhost
rabbitmqctl set_permissions -p myvhost myuser ".*" ".*" ".*"

view raw

rabbitmq.sh

hosted with ❤ by GitHub

3) Install Celery (django-celery is only necessary if you want to use the database result backend or the Django periodic task admin)


# will also install: billiard python-dateutil kombu anyjson amqp
sudo pip install Celery
# install the Django Celery lib
sudo pip install django-celery

view raw

celery.sh

hosted with ❤ by GitHub

4) Configure your Django application (only necessary if using django-celery, be sure to also add django-celery to your installed apps)


CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'

view raw

settings.py

hosted with ❤ by GitHub

5) Create an instance of the celery library for Django to use


from __future__ import absolute_import
import os
from celery import Celery
from django.conf import settings
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'conf.settings')
app = Celery('app.celery')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(settings.INSTALLED_APPS, related_name='tasks')

view raw

celery.py

hosted with ❤ by GitHub

In my application structure I’ve placed this in a file called celery.py:

6) Sync your DB (only necessary if using django-celery)


# when using South
python manage.py migrate djcelery
# when not using South
python manage.py syncdb

view raw

manage.py

hosted with ❤ by GitHub

You can now run Celery using the following commands:

$ celery -A app worker -l info

$ celery beat -A app

Getting Celery Running with Django, django-celery, RabbitMQ and MacPorts

UPDATE: With the release of Celery 3.1, these instructions are out of date. I’ll be posting some notes on migration and doing a fresh 3.1 installation in the near future.

I recently needed an asynchronous task queue and found the fantastic Celery Project to be just what I was looking for.

Getting up and running was fairly simple as there’s lots of great documentation…

Django + Celery
http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html

RabbitMQ
http://docs.celeryproject.org/en/latest/getting-started/brokers/rabbitmq.html
http://www.rabbitmq.com/install-macports.html

But rather than read all of those, here’s a quick rundown of the 5 steps it takes to get up and running…

1) Install RabbitMQ


# required ports
sudo port install erlang +ssl
sudo port install rabbitmq-server
# start the server on system boot
sudo port load rabbitmq-server
# OPTIONAL – install the librabbitmq C library (it's faster thant the default amqp)
sudo pip install librabbitmq

view raw

installs.sh

hosted with ❤ by GitHub

2) OPTIONAL – configure a rabbitmq user


# RabbitMQ installs with a default username / password of guest / guest
# you can change that by creating a new user
rabbitmqctl add_user myuser mypassword
rabbitmqctl add_vhost myvhost
rabbitmqctl set_permissions -p myvhost myuser ".*" ".*" ".*"

view raw

rabbitmq.sh

hosted with ❤ by GitHub

3) Install Celery


# will also install: billiard python-dateutil kombu anyjson amqp
sudo pip install Celery
# install the Django Celery lib
sudo pip install django-celery

view raw

celery.sh

hosted with ❤ by GitHub

4) Configure your Django application


# add the following to your settings.py
import djcelery
djcelery.setup_loader()
# assuming you didn't change the default RabbitMQ username / password
BROKER_URL = 'amqp://guest:guest@localhost:5672/'

view raw

settings.py

hosted with ❤ by GitHub

5) Sync your DB


# when using South
python manage.py migrate djcelery
# when not using South
python manage.py syncdb

view raw

manage.py

hosted with ❤ by GitHub

Boom goes the dynamite. You can now jump into the beginners Django tutorial found here.