Django and MySQL + Emoticons

This has been documented elsewhere previously but for my own recollection, here it is again.

If you create a typical UTF-8 database (CHARACTER SET utf8 COLLATE utf8_unicode_ci), you’ll run into the following error if you attempt to save 4-byte characters to MySQL (i.e. emoticons). This is due to MySQL’s 3 byte limit on utf-8 characters.

To remedy this issue, you’ll need to make a couple of configuration changes:

1) Switch your MySQL database to the utf8mb4 character set (you’ll need MySQL 5.5 or later).

2) Update your Django database settings to use the utf8mb4 encoding:

DATABASES = {
'default': {
'ENGINE':'django.db.backends.mysql',
'OPTIONS': {'charset': 'utf8mb4'},
}
}

view raw
settings.py
hosted with ❤ by GitHub

One thing to watch out for, if you have a CharField with a max_length of 255 characters and it has an index on it (i.e. unique), you’ll need to reduce the max_length to 191 as utf8mb4 takes up 33% more space. More info can be found in this Django ticket.

Django Form Credit Card Field with Pattern, Length and Luhn Validation

I’ve been doing a bit of ecommerce work lately and one of my needs was a credit card field for a Django form. Ideally it needed to support all major cards (PayPal offers a good reference list) and be easily extendable in the future.

All I could find via Google was this older Django project and various blog posts, none of which really fit my needs.

After a bit more digging I decided to port over portions of Stripe’s jQuery.payment which does almost exactly what I needed.

from django import forms
from django.forms.widgets import TextInput
from django.utils.translation import ugettext_lazy as _
class TelephoneInput(TextInput):
# switch input type to type tel so that the numeric keyboard shows on mobile devices
input_type = 'tel'
class CreditCardField(forms.CharField):
# validates almost all of the example cards from PayPal
# https://www.paypalobjects.com/en_US/vhelp/paypalmanager_help/credit_card_numbers.htm
cards = [
{
'type': 'maestro',
'patterns': [5018, 502, 503, 506, 56, 58, 639, 6220, 67],
'length': [12, 13, 14, 15, 16, 17, 18, 19],
'cvvLength': [3],
'luhn': True
}, {
'type': 'forbrugsforeningen',
'patterns': [600],
'length': [16],
'cvvLength': [3],
'luhn': True
}, {
'type': 'dankort',
'patterns': [5019],
'length': [16],
'cvvLength': [3],
'luhn': True
}, {
'type': 'visa',
'patterns': [4],
'length': [13, 16],
'cvvLength': [3],
'luhn': True
}, {
'type': 'mastercard',
'patterns': [51, 52, 53, 54, 55, 22, 23, 24, 25, 26, 27],
'length': [16],
'cvvLength': [3],
'luhn': True
}, {
'type': 'amex',
'patterns': [34, 37],
'length': [15],
'cvvLength': [3, 4],
'luhn': True
}, {
'type': 'dinersclub',
'patterns': [30, 36, 38, 39],
'length': [14],
'cvvLength': [3],
'luhn': True
}, {
'type': 'discover',
'patterns': [60, 64, 65, 622],
'length': [16],
'cvvLength': [3],
'luhn': True
}, {
'type': 'unionpay',
'patterns': [62, 88],
'length': [16, 17, 18, 19],
'cvvLength': [3],
'luhn': False
}, {
'type': 'jcb',
'patterns': [35],
'length': [16],
'cvvLength': [3],
'luhn': True
}
]
def __init__(self, placeholder=None, *args, **kwargs):
super(CreditCardField, self).__init__(
# override default widget
widget=TelephoneInput(attrs={
'placeholder': placeholder
})
, *args, **kwargs)
default_error_messages = {
'invalid': _(u'The credit card number is invalid'),
}
def clean(self, value):
# ensure no spaces or dashes
value = value.replace(' ', '').replace('-', '')
# get the card type and its specs
card = self.card_from_number(value)
# if no card found, invalid
if not card:
raise forms.ValidationError(self.error_messages['invalid'])
# check the length
if not len(value) in card['length']:
raise forms.ValidationError(self.error_messages['invalid'])
# test luhn if necessary
if card['luhn']:
if not self.validate_mod10(value):
raise forms.ValidationError(self.error_messages['invalid'])
return value
def card_from_number(self, num):
# find this card, based on the card number, in the defined set of cards
for card in self.cards:
for pattern in card['patterns']:
if (str(pattern) == str(num)[:len(str(pattern))]):
return card
def validate_mod10(self, num):
# validate card number using the Luhn (mod 10) algorithm
checksum, factor = 0, 1
for c in reversed(num):
for c in str(factor * int(c)):
checksum += int(c)
factor = 3 factor
return checksum % 10 == 0

view raw
fields.py
hosted with ❤ by GitHub

You can then use this field in your form:

class CreditCardForm(ModelForm):
card_number = CreditCardField(placeholder=u'0000 0000 0000 0000', min_length=12, max_length=19)

view raw
form.py
hosted with ❤ by GitHub

Switching from django-storages to django-storages-redux

django-storages provides a variety of storage backends in a single library. Unfortunately it hasn’t seen a release since March of 2013 despite widespread usage and support for the library.

django-storages-redux is a Python 3 & Django 1.8+ compatible fork of the original library that’s thankfully seeing lots of ongoing maintenance and updates.

Switching over was fairly painless, with just a couple method signatures needing updates.

I highly recommend switching over if you haven’t already.

Things to Consider when Upgrading to Django 1.8

Django 1.8 was released back on April 1 and there’s a few things to be aware of when making the upgrade…

1) django.contrib.formtools has been removed. If you were making use of it, grab the new 3rd party library.

2) A good chunk of the django-secure third-party library has been integrated into Django as part of the new django.middleware.security.SecurityMiddleware. Read up on how to configure the new settings.

3) Django now supports multiple template engines with built-in support for the Django template language and for Jinja2. As part of this change you’ll need to update your template settings (for now Django will still use your existing settings, but they are deprecated and will go away with a future release).

4) Django Compressor was incompatible with Django > 1.7. This is now rectified with the recent release of version 1.5.

As always, definitely read the release notes as there’s lots of new stuff along with minor changes and bug fixes in this release.

Create or Update with a Django ModelForm

I recently had a use case where I needed to have a single form that could act as either a create or update form. The tricky part was that I wouldn’t know which one was necessary until the data was submitted by the user.

To solve this, I started with Django’s generic UpdateView and overwrote the get_object method so that it would either get the existing record or create a new one (using get_or_create) based on the data being submitted. In my case (and the example below), I had two values that determined whether a new record was necessary or if an existing one should be used.

The underlying ModelForm didn’t require any modification (you could even just use Django’s automatically generated ModelForm by specifying the model in your view).

Here’s the code:

from django.views.generic import UpdateView
from forms import MyModelForm
from models import MyModel
class MyUpdateView(UpdateView):
# specify a custom ModelForm
form_class = MyModelForm
# or simply specify the Model
# model = MyModel
def get_object(self, queryset=None):
# get the existing object or created a new one
obj, created = MyModel.objects.get_or_create(col_1=self.kwargs['value_1'], col_2=self.kwargs['value_2'])
return obj

view raw
view.py
hosted with ❤ by GitHub

Sort Django Query (Order By) Using Values Within IN()

In MySQL, you can use the FIELD() function to easily sort a result set by a list of ordered ids:

SELECT id, name
FROM table
WHERE name IN (9, 8, 1, 2, 7, 3)
ORDER BY FIELD(id, 9, 8, 1, 2, 7, 3)

view raw
query.sql
hosted with ❤ by GitHub

To accomplish this in Django, you can make use of the extra() QuerySet method to create an additional field in the SELECT statement which can then be using for sorting in the FIELD method.

ids = [9, 8, 1, 2, 7, 3]
results = Model.objects.filter(id__in=ids).extra(
select={'manual': 'FIELD(id,%s)' % ','.join(map(str, ids))},
order_by=['manual']
)

view raw
query.py
hosted with ❤ by GitHub

Combine 2 Django Querysets from Different Models

If you’ve ever tried to concatenating two or more querysets from different models (i.e. combined = queryset1 | queryset2), you’ve hit this lovely error:

Cannot combine queries on two different base models.

The solution to this is to use itertools.

from itertools import chain
result_list = list(chain(queryset1, queryset2))

view raw
query.py
hosted with ❤ by GitHub

This allows you to not only combine the querysets into a single iterable, but it also allows you to sort the entire set by a shared field such as the date created:

from itertools import chain
from operator import attrgetter
# ascending oreder
result_list = sorted(
chain(queryset1, queryset2),
key=attrgetter('date_created'))
# descending order
result_list = sorted(
chain(queryset1, queryset2),
key=attrgetter('date_created'),
reverse=True)

view raw
query.py
hosted with ❤ by GitHub

Django Built-In Template Tags in Views

Django’s built-in template tags are simply Python functions which means they can be used in places other than templates such as class-based views.

Here’s an example using pluralize:

from django.views.generic import ListView
from django.template.defaultfilters import pluralize
from myapp.models.Product
class MyView(ListView):
queryset = Product.objects.all()
template_name = 'product.html'
context_object_name = 'products'
def get_context_data(self, **kwargs):
context = super(MyView, self).get_context_data(**kwargs)
context['title'] = len(context['products']) + ' Product' + pluralize(context['products']) + ' Found'
return context

view raw
view.py
hosted with ❤ by GitHub

Django, Haystack and Elasticsearch – Part 1

I’m wrapping up a little side project at the moment (more on that very soon) which required full-text search, autocomplete, and a few other bits of search related functionality.

After some research I landed upon the combination of Elasticsearch and the awesome Django application Haystack.

First step was to get Elasticsearch up and running locally on OS X…

1) Download latest zip from http://www.elasticsearch.org/overview/elkdownloads/. A good spot is:

/opt/elasticsearch-1.1.x

2) Create the following directories:

/opt/elasticsearch-1.1.x/data
/opt/elasticsearch-1.1.x/work
/opt/elasticsearch-1.1.x/logs

3) Add the following to your .profile (allows you to run Elasticsearch from the command prompt without the full path):

# elasticsearch
export ES_HOME=/opt/elasticsearch-1.1.x
PATH=$ES_HOME/bin:$PATH

view raw
.profile
hosted with ❤ by GitHub

4) Update the following values in the Elasticsearch config file:

# /opt/elasticsearch-1.1.x/config/elasticsearch.yml
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["127.0.0.1"]
cluster.name: elasticsearch
network.host: 127.0.0.1
http.port: 9200
path.conf: /opt/elasticsearch-1.1.x/config
path.data: /opt/elasticsearch-1.1.x/data
path.work: /opt/elasticsearch-1.1.x/work
path.logs: /opt/elasticsearch-1.1.x/logs

view raw
elasticsearch.yml
hosted with ❤ by GitHub

5) Ensure all requirements are installed (django-haystack, pyelasticsearch, requests, simplejson):

pip install django-haystack
pip install pyelasticsearch

view raw
install.sh
hosted with ❤ by GitHub

6) You should now be able to start Elasticsearch:

$ elasticsearch
[2014-05-14 08:15:05,257][INFO ][node ] [Aminedi] version[1.1.1], pid[46224], build[f1585f0/2014-04-16T14:27:12Z]
[2014-05-14 08:15:05,258][INFO ][node ] [Aminedi] initializing …
[2014-05-14 08:15:05,271][INFO ][plugins ] [Aminedi] loaded [], sites []
[2014-05-14 08:15:07,136][INFO ][node ] [Aminedi] initialized
[2014-05-14 08:15:07,136][INFO ][node ] [Aminedi] starting …
[2014-05-14 08:15:07,211][INFO ][transport ] [Aminedi] bound_address {inet[/127.0.0.1:9300]}, publish_address {inet[/127.0.0.1:9300]}
[2014-05-14 08:15:10,262][INFO ][cluster.service ] [Aminedi] new_master [Aminedi][X4diAes4TrOMTk4eAdbhnA][mbp.home][inet[/127.0.0.1:9300]], reason: zen-disco-join (elected_as_master)
[2014-05-14 08:15:10,284][INFO ][discovery ] [Aminedi] elasticsearch/X4diAes4TrOMTk4eAdbhnA
[2014-05-14 08:15:10,298][INFO ][http ] [Aminedi] bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[/127.0.0.1:9200]}
[2014-05-14 08:15:10,784][INFO ][gateway ] [Aminedi] recovered [1] indices into cluster_state
[2014-05-14 08:15:10,785][INFO ][node ] [Aminedi] started

view raw
elasticsearch.sh
hosted with ❤ by GitHub

7) Add Haystack to your Django config:

# add to installed apps
INSTALLED_APPS = (
'haystack',
)
# haystack search using elasticsearch
HAYSTACK_CONNECTIONS = {
'default': {
'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
'URL': 'http://127.0.0.1:9200/',
'INDEX_NAME': 'haystack',
},
}
# http://django-haystack.readthedocs.org/en/latest/signal_processors.html
HAYSTACK_SIGNAL_PROCESSOR = 'haystack.signals.RealtimeSignalProcessor'
# increase the default number of results (from 20)
HAYSTACK_SEARCH_RESULTS_PER_PAGE = 40

view raw
config.py
hosted with ❤ by GitHub

8) After you’ve added your search indexes, you can use manage.py to rebuild the search index:

$ python manage.py rebuild_index

Django Template Tag for Truncating Characters at a Word Boundary

The default truncatechars template tag truncates a string if it is longer than the specified number of characters but does so exactly at the character count, irrespective of whether it’s the middle of a word or not.

Here’s a smarter version that clips the text at the word boundary:

from django import template
register = template.Library()
# truncate chars but leaving last word complete
@register.filter(name='smarttruncatechars')
def smart_truncate_chars(value, max_length):
if len(value) > max_length:
# limits the number of characters in value to max_length (blunt cut)
truncd_val = value[:max_length]
# check if the next upcoming character after the limit is not a space,
# in which case it might be a word continuing
if value[max_length] != ' ':
# rfind will return the last index where matching the searched character,
# in this case we are looking for the last space
# then we only return the number of character up to that last space
truncd_val = truncd_val[:truncd_val.rfind(' ')]
return truncd_val + '…'
return value