Installing Cryptography via Pip with MacPorts or Homebrew

If you have tried installing / updating pyOpenSSL to version 0.14 you may have run into an issue when it tries to install the required package cryptography. The specific error I encountered was:

raise ffiplatform.VerificationError(error)
cffi.ffiplatform.VerificationError: importing '/private/tmp/pip_build_root/cryptography/cryptography/hazmat/bindings/__pycache__/_cffi__x5eaa210axf0ae7e21.so': dlopen(/private/tmp/pip_build_root/cryptography/cryptography/hazmat/bindings/__pycache__/_cffi__x5eaa210axf0ae7e21.so, 2): Symbol not found: _CRYPTO_malloc_debug_init
Referenced from: /private/tmp/pip_build_root/cryptography/cryptography/hazmat/bindings/__pycache__/_cffi__x5eaa210axf0ae7e21.so
Expected in: flat namespace
in /private/tmp/pip_build_root/cryptography/cryptography/hazmat/bindings/__pycache__/_cffi__x5eaa210axf0ae7e21.so

view raw
error.sh
hosted with ❤ by GitHub

To get around this, you’ll need to set some environment variables before installing cryptography:

# install via MacPorts
sudo port install openssl
sudo env ARCHFLAGS="-arch x86_64" LDFLAGS="-L/opt/local/lib" CFLAGS="-I/opt/local/include" pip install cryptography
# install via Homebrew
brew install openssl
env ARCHFLAGS="-arch x86_64" LDFLAGS="-L/usr/local/opt/openssl/lib" CFLAGS="-I/usr/local/opt/openssl/include" pip install cryptography

view raw
install.sh
hosted with ❤ by GitHub

A pull request has been opened to add this to the documentation but as of this writing it has not yet been merged.

PyCharm Command Not Found When Using Virtualenv & MacPorts

I recently switched a project over to using virtualenv for my PyCharm Django server run/debug configuration and ran into a ‘command not found’ error when trying to use another MacPorts installed package (sass) from Django (via Django Compressor).

Here is my original server configuration in PyCharm:

A quick look at the PATH environment variable identified the trouble:

MacPorts Python (/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7)

>>> print os.environ['PATH']
/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin

Virtualenv Python (/Users/chriskief/.virtualenvs/exampleproject/bin/python)

>>> print os.environ['PATH']
/usr/bin:/bin:/usr/sbin:/sbin:/Users/chriskief/.virtualenvs/exampleproject/bin

The MacPorts bin and sbin are missing from the PATH, and that’s where sass lives:

which sass
/opt/local/bin/sass

The solution is to add the MacPorts directories (/opt/local/bin:/opt/local/sbin:$PATH) to your environment variables’ PATH in the run/debug configuration:

As a side note, if you run the Django server using the virtualenv from your terminal, you won’t encounter this error because MacPorts includes these directories in your .profile PATH:

# MacPorts Installer addition on 2013-10-26_at_16:30:30: adding an appropriate PATH variable for use with MacPorts.
export PATH=/opt/local/bin:/opt/local/sbin:$PATH
# Finished adapting your PATH environment variable for use with MacPorts.

MacPorts, Mavericks & MySQL 5.6 with Memcached

If you’ve upgraded to Mavericks you’ve probably realized that MacPorts MySQL 5.6 would not build due to some issues with MySQL itself.

That issue has now been fixed with MySQL version 5.6.15 and this changeset which is now live in the port index. Simply install like usual:

sudo port install mysql56-server

One thing the portfile doesn’t contain is the flag to enable the new 5.6 InnoDB Memcached Plugin. If you’d like to enable it, you’ll need to create a local portfile with the following changes:

# change
name                mysql56
# to
name                mysql56-custom


# change
-DWITH_SSL:STRING=bundled
# to
-DWITH_SSL:STRING=bundled \
-DWITH_INNODB_MEMCACHED=ON

If you’ve never worked with local portfiles before, here’s a quick tutorial…

# the following assumes a typical MacPorts install to /opt/local
# create a directory to hold local ports
sudo mkdir /opt/custom
# edit /opt/local/etc/macports/sources.conf and add the following line before rsync://rsync.macports.org/release/tarballs/ports.tar
# file:///opt/custom
# create the directories to hold the port file
sudo mkdir -p /opt/custom/databases/mysql56/files/
# grab the original portfile and save it to /opt/custom/databases/mysql56/Portfile
# https://trac.macports.org/browser/trunk/dports/databases/mysql56/Portfile
# also grab the 5 supporting files and save them in /opt/custom/databases/mysql56/files/
# https://trac.macports.org/browser/trunk/dports/databases/mysql56/files
# update the Portfile with the following changes:
# Line 6:
# OLD: name mysql56
# NEW: name mysql56-custom
# Line 97:
# OLD: -DWITH_SSL:STRING=bundled
# NEW: -DWITH_SSL:STRING=bundled \
# -DWITH_INNODB_MEMCACHED=ON
# index the new port
cd /opt/custom
sudo portindex
# verify the port was added
port search mysql56-custom
# you should see:
# mysql56-custom @5.6.15 (databases)
# mysql56-custom-server @5.6.15
# install
sudo port install mysql56-custom-server

view raw
localport.sh
hosted with ❤ by GitHub

Mavericks, MacPorts, PostgreSQL 9, Tomcat 6 and PostgreSQL Studio

Now that Amazon Web Services is supporting PostgreSQL I figured it’s about time I got around to getting it setup locally.

First step was to install PostgreSQL via MacPorts:

# install and select
sudo port install postgresql93-server
sudo port select –set postgresql postgresql93
# load at startup
sudo port load postgresql93-server
# create the default database
sudo mkdir -p /opt/local/var/db/postgresql93/defaultdb
sudo chown postgres:postgres /opt/local/var/db/postgresql93/defaultdb
sudo su postgres -c '/opt/local/lib/postgresql93/bin/initdb -D /opt/local/var/db/postgresql93/defaultdb'
# create a new user (or you can simply use the default user postgres which does not require a password)
createuser –superuser your_username -U postgres -P
# create a new database
createdb database_name

view raw
postgresql.sh
hosted with ❤ by GitHub

You should now be able to connect to your database using pgAdmin or a similar tool. If you’re having trouble, a quick reboot of your machine should get it working (assuming you set PostgreSQL to load at startup).

I also wanted to try out PostgresSQL Studio which requires Tomcat so that got installed next:

# install
sudo port install tomcat6
# add -Djava.awt.headless=true to /opt/local/share/java/tomcat6/bin/tomcatctl (line 65)
JAVA_OPTS="$JAVA_OPTS -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true"
# without this you’ll get this error in /opt/local/share/java/tomcat6/logs/catalina.err:
# 2013-11-16 23:39:17.389 jsvc[1698:203] Apple AWT Java VM was loaded on first thread — can't start AWT.
# Nov 16, 2013 11:39:17 PM org.apache.catalina.startup.Bootstrap initClassLoaders
# SEVERE: Class loader creation threw exception
# java.lang.InternalError: Can't start the AWT because Java was started on the first thread. Make sure StartOnFirstThread is not specified in your application's Info.plist or on the command line
# load at startup
sudo port load tomcat6

view raw
tomcat.sh
hosted with ❤ by GitHub

Installing Tomcat led to a momentary headache as it would not start, throwing the following error – Apple AWT Java VM was loaded on first thread — can’t start AWT. All of the OS X / MacPorts / Tomcat instructions on Google were pretty out of date so it took a bit of digging to figure out what was going on. Luckily the fix was rather simple – just edit tomcatctl with the change shown in the Gist above.

The final step was to download PostgreSQL Studio, unzip the file and drop pgstudio.war into /opt/local/share/java/tomcat6/webapps/.

After a few moments Tomcat will autodeploy the war and you’ll be able to browse the application at http://localhost:8080/pgstudio/.

Celery 3.1 with Django, django-celery, RabbitMQ and MacPorts

Celery 3.1 was released earlier this week and with it comes Django support out of the box. This means Django users can now use the Celery API directly rather than depending on django-celery (the libary is still needed if you want to make use of the database result backend or the Django periodic task admin). This also means you can now use the celery command directly rather than going through manage.py.

In this post I’ll cover the changes necessary to migrate to 3.1 from a previous installation as well as provide updated instructions for getting started from scratch. First, the migration…

1) Remove django-celery integration (even if you are using django-celery for the database result backend or the periodic task admin, you still need to do this). Remove the following from your Django settings (also remove from wsgi.py if present there):

import djcelery
djcelery.setup_loader()

view raw
settings.py
hosted with ❤ by GitHub

2) If you’re using the default RabbitMQ broker (and default username / password), you won’t need this in your settings either:

BROKER_URL = 'amqp://guest:guest@localhost:5672//'

view raw
settings.py
hosted with ❤ by GitHub

3) To still use the database result backend from django-celery, ensure the following is in your Django settings:

CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'

view raw
settings.py
hosted with ❤ by GitHub

4) Create an instance of the celery library for Django to use.

from __future__ import absolute_import
import os
from celery import Celery
from django.conf import settings
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'conf.settings')
app = Celery('app.celery')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(settings.INSTALLED_APPS, related_name='tasks')

view raw
celery.py
hosted with ❤ by GitHub

This code performs a number of functions:

  • sets the default DJANGO_SETTINGS_MODULE for the celery command-line program
  • creates an instance of the celery library
  • adds the Django settings module as a configuration source for Celery (allows you to configure Celery directly from your Django settings)
  • autodiscovers tasks within your installed apps

In my application structure I’ve placed this in a file called celery.py:

5) If you want to make use of the shared_task decorator as demonstrated in the Django example application, add the following to your application’s __init__.py:

from __future__ import absolute_import
# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app

view raw
__init__.py
hosted with ❤ by GitHub

This decorator returns a proxy that always points to the currently active Celery instance, allowing you to define tasks within multiple applications and still use a single Celery instance.

That’s it for changes. Now the final piece is making use of new commands for running Celery. To run a worker, you previously used the following:

$ python manage.py celery worker --loglevel=info

This is replaced with:

$ celery -A app worker -l info

Similarly, running Celery beat:

$ python manage.py celery beat

Is now:

$ celery beat -A app

Note that each command requires the name of your Django application (the one that contains celery.py) to be passed via -A.


For a fresh install, here’s a quick rundown of the 6 steps it takes to get up and running…

1) Install RabbitMQ

# required ports
sudo port install erlang +ssl
sudo port install rabbitmq-server
# start the server on system boot
sudo port load rabbitmq-server
# OPTIONAL – install the librabbitmq C library (it's faster thant the default amqp)
sudo pip install librabbitmq

view raw
installs.sh
hosted with ❤ by GitHub

2) OPTIONAL – configure a rabbitmq user

# RabbitMQ installs with a default username / password of guest / guest
# you can change that by creating a new user
rabbitmqctl add_user myuser mypassword
rabbitmqctl add_vhost myvhost
rabbitmqctl set_permissions -p myvhost myuser ".*" ".*" ".*"

view raw
rabbitmq.sh
hosted with ❤ by GitHub

3) Install Celery (django-celery is only necessary if you want to use the database result backend or the Django periodic task admin)

# will also install: billiard python-dateutil kombu anyjson amqp
sudo pip install Celery
# install the Django Celery lib
sudo pip install django-celery

view raw
celery.sh
hosted with ❤ by GitHub

4) Configure your Django application (only necessary if using django-celery, be sure to also add django-celery to your installed apps)

CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'

view raw
settings.py
hosted with ❤ by GitHub

5) Create an instance of the celery library for Django to use

from __future__ import absolute_import
import os
from celery import Celery
from django.conf import settings
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'conf.settings')
app = Celery('app.celery')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(settings.INSTALLED_APPS, related_name='tasks')

view raw
celery.py
hosted with ❤ by GitHub

In my application structure I’ve placed this in a file called celery.py:

6) Sync your DB (only necessary if using django-celery)

# when using South
python manage.py migrate djcelery
# when not using South
python manage.py syncdb

view raw
manage.py
hosted with ❤ by GitHub

You can now run Celery using the following commands:

$ celery -A app worker -l info

$ celery beat -A app

MacPorts & Mavericks OS X 10.9 & Gnutar Errors

UPDATE – The 10.9 Mavericks binary is now available here.

If you try installing or updating MacPorts ports after upgrading OS X to 10.9, you’ll hit an error with gnutar:

mbp:tar-1.27 chriskief$ sudo port upgrade outdated
> Extracting apr
Error: org.macports.extract for port apr returned: command execution failed
Please see the log file for port apr for details:
/opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_devel_apr/apr/main.log
Error: Unable to upgrade port: 1
To report a bug, follow the instructions in the guide:
http://guide.macports.org/#project.tickets

view raw
macports.sh
hosted with ❤ by GitHub

:debug:extract Assembled command: 'cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_devel_apr/apr/work" && /usr/bin/bzip2 -dc '/opt/local/var/macports/distfiles/apr/apr-1.4.8.tar.bz2' | /usr/bin/gnutar –no-same-owner -xf -'
:debug:extract Executing command line: cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_devel_apr/apr/work" && /usr/bin/bzip2 -dc '/opt/local/var/macports/distfiles/apr/apr-1.4.8.tar.bz2' | /usr/bin/gnutar –no-same-owner -xf –
:info:extract sh: /usr/bin/gnutar: No such file or directory

view raw
macports.log
hosted with ❤ by GitHub

The recommended fix from MacPorts is to reinstall MacPorts and all ports after a major OS upgrade. Unfortunately there is no binary for Mavericks yet and quite a few ports are failing to build right now including MySQL 5.6 and Python 2.7.

The  workaround is to keep your current MacPorts installation and install gnutar from source:

wget http://ftp.gnu.org/gnu/tar/tar-1.27.tar.gz
tar xzvf tar-1.27.tar.gz
cd tar-1.27
./configure
make
sudo mv ./src/tar /usr/bin/gnutar

view raw
install.sh
hosted with ❤ by GitHub

This is definitely not a long term solution and should really just be used for new ports that are must haves. I wouldn’t run port upgrade outdated or try to upgrade any specific ports as you will find that not all ports will build.

Once a Maverick’s binary is out, be sure to remove this gnutar before doing the migration.

Read more discussion of MacPorts on Mavericks.

Note: Be sure to build and install gnutar from source and not just symlink…

Getting Celery Running with Django, django-celery, RabbitMQ and MacPorts

UPDATE: With the release of Celery 3.1, these instructions are out of date. I’ll be posting some notes on migration and doing a fresh 3.1 installation in the near future.


I recently needed an asynchronous task queue and found the fantastic Celery Project to be just what I was looking for.

Getting up and running was fairly simple as there’s lots of great documentation…

Django + Celery
http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html

RabbitMQ
http://docs.celeryproject.org/en/latest/getting-started/brokers/rabbitmq.html
http://www.rabbitmq.com/install-macports.html

But rather than read all of those, here’s a quick rundown of the 5 steps it takes to get up and running…

1) Install RabbitMQ

# required ports
sudo port install erlang +ssl
sudo port install rabbitmq-server
# start the server on system boot
sudo port load rabbitmq-server
# OPTIONAL – install the librabbitmq C library (it's faster thant the default amqp)
sudo pip install librabbitmq

view raw
installs.sh
hosted with ❤ by GitHub

2) OPTIONAL – configure a rabbitmq user

# RabbitMQ installs with a default username / password of guest / guest
# you can change that by creating a new user
rabbitmqctl add_user myuser mypassword
rabbitmqctl add_vhost myvhost
rabbitmqctl set_permissions -p myvhost myuser ".*" ".*" ".*"

view raw
rabbitmq.sh
hosted with ❤ by GitHub

3) Install Celery

# will also install: billiard python-dateutil kombu anyjson amqp
sudo pip install Celery
# install the Django Celery lib
sudo pip install django-celery

view raw
celery.sh
hosted with ❤ by GitHub

4) Configure your Django application

# add the following to your settings.py
import djcelery
djcelery.setup_loader()
# assuming you didn't change the default RabbitMQ username / password
BROKER_URL = 'amqp://guest:guest@localhost:5672/'

view raw
settings.py
hosted with ❤ by GitHub

5) Sync your DB

# when using South
python manage.py migrate djcelery
# when not using South
python manage.py syncdb

view raw
manage.py
hosted with ❤ by GitHub

Boom goes the dynamite. You can now jump into the beginners Django tutorial found here.

Update MacPorts, Ruby Gems, Node Packages and Python Packages with One Command

Off topic but still helpful…

Like many devs I make use of a variety of open source software, package management tools, etc – all of which need to be updated on a regular basis to stay up-to-date with the most recent releases.

Rather than manage each one individually, a simple alias in your ~/.profile comes in very handy:

alias getuptodate='sudo port selfupdate; sudo port upgrade outdated; sudo port uninstall inactive; sudo gem update –system; sudo gem update; sudo npm update uglify-js -g; sudo pip freeze –local | grep -v '^\-e' | cut -d = -f 1 | xargs sudo pip install -U;'

view raw
update.sh
hosted with ❤ by GitHub

Notes:

  • I haven’t been able to find a command to update all installed Node packages at once so I’m listing them individually for now.
  • This assumes that npm is installed via MacPorts.
  • Unfortunately this fails when pip freeze lists a package that can’t be installed from PyPI anymore.

Then you can simply invoke it like so…

$ getuptodate
Password:
> Updating MacPorts base sources using rsync
MacPorts base version 2.1.2 installed,
MacPorts base version 2.1.2 downloaded.
> Updating the ports tree

view raw
run.sh
hosted with ❤ by GitHub

Of course if you need to handle specific versions of a port or gem individually, this won’t work for you. But if you just want to be current on everything, this will do the trick.