Category: Hacking

Django and MySQL + Emoticons

This has been documented elsewhere previously but for my own recollection, here it is again.

If you create a typical UTF-8 database (CHARACTER SET utf8 COLLATE utf8_unicode_ci), you’ll run into the following error if you attempt to save 4-byte characters to MySQL (i.e. emoticons). This is due to MySQL’s 3 byte limit on utf-8 characters.

To remedy this issue, you’ll need to make a couple of configuration changes:

1) Switch your MySQL database to the utf8mb4 character set (you’ll need MySQL 5.5 or later).

2) Update your Django database settings to use the utf8mb4 encoding:

One thing to watch out for, if you have a CharField with a max_length of 255 characters and it has an index on it (i.e. unique), you’ll need to reduce the max_length to 191 as utf8mb4 takes up 33% more space. More info can be found in this Django ticket.

Django Form Credit Card Field with Pattern, Length and Luhn Validation

I’ve been doing a bit of ecommerce work lately and one of my needs was a credit card field for a Django form. Ideally it needed to support all major cards (PayPal offers a good reference list) and be easily extendable in the future.

All I could find via Google was this older Django project and various blog posts, none of which really fit my needs.

After a bit more digging I decided to port over portions of Stripe’s jQuery.payment which does almost exactly what I needed.

You can then use this field in your form:

Invalid Certificate after Security Update 2015-004 in Mavericks

After recently installing Security Update 2015-004, I found that I could no longer browse to any website using the root certificate “VeriSign Class 3 Public Primary Certification Authority – G5” without a security warning (“invalid certificate”). This included sites such as Twitter and Apple, and it also meant that applications such as Software Update would no longer function.

After digging into it (see here, here, here and here) I found the cause was a chain of events that while a bit convoluted, were fairly prevalent among users.

First off, 2015-004 updated the list of trusted root CAs which by itself isn’t an issue. The problem was when I then logged into Amazon S3 using an older version of Cyberduck (< 4.7). That version of Cyberduck was adding the certificate chain retrieved from Amazon to my login keychain which also by itself isn't an issue. The problem was that the intermediate certs Amazon was using were outdated and signed with 1024bits. This caused a mismatch between the certs installed by 2015-004 and the ones being saved to the keychain by Cyberduck. Like I said, convoluted.

Luckily everyone seems to have implemented fixes – Cyberduck no longer writes the intermediate certs to the keychain (as of version 4.7) and Amazon has updated their intermediate certs to 2048bit signatures.

If you run into this issue, you probably still have the invalid certs sitting in your keychain. Simply open up Keychain Access and delete the bogus entries in the login keychain so that the system entries are used instead (select login, then Certificates, you should see them at the bottom of the list – "VeriSign Class 3 Public Primary Certification Authority – G5").

Switching from django-storages to django-storages-redux

django-storages provides a variety of storage backends in a single library. Unfortunately it hasn’t seen a release since March of 2013 despite widespread usage and support for the library.

django-storages-redux is a Python 3 & Django 1.8+ compatible fork of the original library that’s thankfully seeing lots of ongoing maintenance and updates.

Switching over was fairly painless, with just a couple method signatures needing updates.

I highly recommend switching over if you haven’t already.

Things to Consider when Upgrading to Django 1.8

Django 1.8 was released back on April 1 and there’s a few things to be aware of when making the upgrade…

1) django.contrib.formtools has been removed. If you were making use of it, grab the new 3rd party library.

2) A good chunk of the django-secure third-party library has been integrated into Django as part of the new django.middleware.security.SecurityMiddleware. Read up on how to configure the new settings.

3) Django now supports multiple template engines with built-in support for the Django template language and for Jinja2. As part of this change you’ll need to update your template settings (for now Django will still use your existing settings, but they are deprecated and will go away with a future release).

4) Django Compressor was incompatible with Django > 1.7. This is now rectified with the recent release of version 1.5.

As always, definitely read the release notes as there’s lots of new stuff along with minor changes and bug fixes in this release.

Create or Update with a Django ModelForm

I recently had a use case where I needed to have a single form that could act as either a create or update form. The tricky part was that I wouldn’t know which one was necessary until the data was submitted by the user.

To solve this, I started with Django’s generic UpdateView and overwrote the get_object method so that it would either get the existing record or create a new one (using get_or_create) based on the data being submitted. In my case (and the example below), I had two values that determined whether a new record was necessary or if an existing one should be used.

The underlying ModelForm didn’t require any modification (you could even just use Django’s automatically generated ModelForm by specifying the model in your view).

Here’s the code:

Sort Django Query (Order By) Using Values Within IN()

In MySQL, you can use the FIELD() function to easily sort a result set by a list of ordered ids:

To accomplish this in Django, you can make use of the extra() QuerySet method to create an additional field in the SELECT statement which can then be using for sorting in the FIELD method.