Category: Hacking

Django and MySQL + Emoticons

This has been documented elsewhere previously but for my own recollection, here it is again.

If you create a typical UTF-8 database (CHARACTER SET utf8 COLLATE utf8_unicode_ci), you’ll run into the following error if you attempt to save 4-byte characters to MySQL (i.e. emoticons). This is due to MySQL’s 3 byte limit on utf-8 characters.

To remedy this issue, you’ll need to make a couple of configuration changes:

1) Switch your MySQL database to the utf8mb4 character set (you’ll need MySQL 5.5 or later).

2) Update your Django database settings to use the utf8mb4 encoding:

DATABASES = {
'default': {
'ENGINE':'django.db.backends.mysql',
…
'OPTIONS': {'charset': 'utf8mb4'},
}
}

view raw
settings.py
hosted with ❤ by GitHub

One thing to watch out for, if you have a CharField with a max_length of 255 characters and it has an index on it (i.e. unique), you’ll need to reduce the max_length to 191 as utf8mb4 takes up 33% more space. More info can be found in this Django ticket.

Django Form Credit Card Field with Pattern, Length and Luhn Validation

I’ve been doing a bit of ecommerce work lately and one of my needs was a credit card field for a Django form. Ideally it needed to support all major cards (PayPal offers a good reference list) and be easily extendable in the future.

All I could find via Google was this older Django project and various blog posts, none of which really fit my needs.

After a bit more digging I decided to port over portions of Stripe’s jQuery.payment which does almost exactly what I needed.

from django import forms
from django.forms.widgets import TextInput
from django.utils.translation import ugettext_lazy as _
class TelephoneInput(TextInput):
# switch input type to type tel so that the numeric keyboard shows on mobile devices
input_type = 'tel'
class CreditCardField(forms.CharField):
# validates almost all of the example cards from PayPal
# https://www.paypalobjects.com/en_US/vhelp/paypalmanager_help/credit_card_numbers.htm
cards = [
{
'type': 'maestro',
'patterns': [5018, 502, 503, 506, 56, 58, 639, 6220, 67],
'length': [12, 13, 14, 15, 16, 17, 18, 19],
'cvvLength': [3],
'luhn': True
}, {
'type': 'forbrugsforeningen',
'patterns': [600],
'length': [16],
'cvvLength': [3],
'luhn': True
}, {
'type': 'dankort',
'patterns': [5019],
'length': [16],
'cvvLength': [3],
'luhn': True
}, {
'type': 'visa',
'patterns': [4],
'length': [13, 16],
'cvvLength': [3],
'luhn': True
}, {
'type': 'mastercard',
'patterns': [51, 52, 53, 54, 55, 22, 23, 24, 25, 26, 27],
'length': [16],
'cvvLength': [3],
'luhn': True
}, {
'type': 'amex',
'patterns': [34, 37],
'length': [15],
'cvvLength': [3, 4],
'luhn': True
}, {
'type': 'dinersclub',
'patterns': [30, 36, 38, 39],
'length': [14],
'cvvLength': [3],
'luhn': True
}, {
'type': 'discover',
'patterns': [60, 64, 65, 622],
'length': [16],
'cvvLength': [3],
'luhn': True
}, {
'type': 'unionpay',
'patterns': [62, 88],
'length': [16, 17, 18, 19],
'cvvLength': [3],
'luhn': False
}, {
'type': 'jcb',
'patterns': [35],
'length': [16],
'cvvLength': [3],
'luhn': True
}
]
def __init__(self, placeholder=None, *args, **kwargs):
super(CreditCardField, self).__init__(
# override default widget
widget=TelephoneInput(attrs={
'placeholder': placeholder
})
, *args, **kwargs)
default_error_messages = {
'invalid': _(u'The credit card number is invalid'),
}
def clean(self, value):
# ensure no spaces or dashes
value = value.replace(' ', '').replace('-', '')
# get the card type and its specs
card = self.card_from_number(value)
# if no card found, invalid
if not card:
raise forms.ValidationError(self.error_messages['invalid'])
# check the length
if not len(value) in card['length']:
raise forms.ValidationError(self.error_messages['invalid'])
# test luhn if necessary
if card['luhn']:
if not self.validate_mod10(value):
raise forms.ValidationError(self.error_messages['invalid'])
return value
def card_from_number(self, num):
# find this card, based on the card number, in the defined set of cards
for card in self.cards:
for pattern in card['patterns']:
if (str(pattern) == str(num)[:len(str(pattern))]):
return card
def validate_mod10(self, num):
# validate card number using the Luhn (mod 10) algorithm
checksum, factor = 0, 1
for c in reversed(num):
for c in str(factor * int(c)):
checksum += int(c)
factor = 3 – factor
return checksum % 10 == 0

view raw
fields.py
hosted with ❤ by GitHub

You can then use this field in your form:

class CreditCardForm(ModelForm):
card_number = CreditCardField(placeholder=u'0000 0000 0000 0000', min_length=12, max_length=19)

view raw
form.py
hosted with ❤ by GitHub

Counting Lines of Code

Here’s a handy little shell script for counting lines of code (recursively in all directories beneath the current directory) based on the type of file:

find . \( -name '*.py' -o -name '*.coffee' -o -name '*.scss' \) | xargs wc -l

view raw
gistfile1.txt
hosted with ❤ by GitHub

Invalid Certificate after Security Update 2015-004 in Mavericks

After recently installing Security Update 2015-004, I found that I could no longer browse to any website using the root certificate “VeriSign Class 3 Public Primary Certification Authority – G5” without a security warning (“invalid certificate”). This included sites such as Twitter and Apple, and it also meant that applications such as Software Update would no longer function.

After digging into it (see here, here, here and here) I found the cause was a chain of events that while a bit convoluted, were fairly prevalent among users.

First off, 2015-004 updated the list of trusted root CAs which by itself isn’t an issue. The problem was when I then logged into Amazon S3 using an older version of Cyberduck (< 4.7). That version of Cyberduck was adding the certificate chain retrieved from Amazon to my login keychain which also by itself isn't an issue. The problem was that the intermediate certs Amazon was using were outdated and signed with 1024bits. This caused a mismatch between the certs installed by 2015-004 and the ones being saved to the keychain by Cyberduck. Like I said, convoluted.

Luckily everyone seems to have implemented fixes – Cyberduck no longer writes the intermediate certs to the keychain (as of version 4.7) and Amazon has updated their intermediate certs to 2048bit signatures.

If you run into this issue, you probably still have the invalid certs sitting in your keychain. Simply open up Keychain Access and delete the bogus entries in the login keychain so that the system entries are used instead (select login, then Certificates, you should see them at the bottom of the list – "VeriSign Class 3 Public Primary Certification Authority – G5").

Switching from django-storages to django-storages-redux

django-storages provides a variety of storage backends in a single library. Unfortunately it hasn’t seen a release since March of 2013 despite widespread usage and support for the library.

django-storages-redux is a Python 3 & Django 1.8+ compatible fork of the original library that’s thankfully seeing lots of ongoing maintenance and updates.

Switching over was fairly painless, with just a couple method signatures needing updates.

I highly recommend switching over if you haven’t already.

Things to Consider when Upgrading to Django 1.8

Django 1.8 was released back on April 1 and there’s a few things to be aware of when making the upgrade…

1) django.contrib.formtools has been removed. If you were making use of it, grab the new 3rd party library.

2) A good chunk of the django-secure third-party library has been integrated into Django as part of the new django.middleware.security.SecurityMiddleware. Read up on how to configure the new settings.

3) Django now supports multiple template engines with built-in support for the Django template language and for Jinja2. As part of this change you’ll need to update your template settings (for now Django will still use your existing settings, but they are deprecated and will go away with a future release).

4) Django Compressor was incompatible with Django > 1.7. This is now rectified with the recent release of version 1.5.

As always, definitely read the release notes as there’s lots of new stuff along with minor changes and bug fixes in this release.

Create or Update with a Django ModelForm

I recently had a use case where I needed to have a single form that could act as either a create or update form. The tricky part was that I wouldn’t know which one was necessary until the data was submitted by the user.

To solve this, I started with Django’s generic UpdateView and overwrote the get_object method so that it would either get the existing record or create a new one (using get_or_create) based on the data being submitted. In my case (and the example below), I had two values that determined whether a new record was necessary or if an existing one should be used.

The underlying ModelForm didn’t require any modification (you could even just use Django’s automatically generated ModelForm by specifying the model in your view).

Here’s the code:

from django.views.generic import UpdateView
from forms import MyModelForm
from models import MyModel
class MyUpdateView(UpdateView):
# specify a custom ModelForm
form_class = MyModelForm
# or simply specify the Model
# model = MyModel
def get_object(self, queryset=None):
# get the existing object or created a new one
obj, created = MyModel.objects.get_or_create(col_1=self.kwargs['value_1'], col_2=self.kwargs['value_2'])
return obj

view raw
view.py
hosted with ❤ by GitHub

Sort Django Query (Order By) Using Values Within IN()

In MySQL, you can use the FIELD() function to easily sort a result set by a list of ordered ids:

SELECT id, name
FROM table
WHERE name IN (9, 8, 1, 2, 7, 3)
ORDER BY FIELD(id, 9, 8, 1, 2, 7, 3)

view raw
query.sql
hosted with ❤ by GitHub

To accomplish this in Django, you can make use of the extra() QuerySet method to create an additional field in the SELECT statement which can then be using for sorting in the FIELD method.

ids = [9, 8, 1, 2, 7, 3]
results = Model.objects.filter(id__in=ids).extra(
select={'manual': 'FIELD(id,%s)' % ','.join(map(str, ids))},
order_by=['manual']
)

view raw
query.py
hosted with ❤ by GitHub