How to make Django slugify work properly with Unicode strings?

Question

What can I do to prevent slugify filter from stripping out non-ASCII alphanumeric characters? (I'm using Django 1.0.2)

cnprog.com has Chinese characters in question URLs, so I looked in their code. They are not using slugify in templates, instead they're calling this method in Question model to get permalinks

def get_absolute_url(self):     return '%s%s' % (reverse('question', args=[self.id]), self.title) 

Are they slugifying the URLs or not?

Answer 1

There is a python package called unidecode that I've adopted for the askbot Q&A forum, it works well for the latin-based alphabets and even looks reasonable for greek:

>>> import unidecode >>> from unidecode import unidecode >>> unidecode(u'διακριτικός') 'diakritikos' 

It does something weird with asian languages:

>>> unidecode(u'影師嗎') 'Ying Shi Ma ' >>>  

Does this make sense?

In askbot we compute slugs like so:

from unidecode import unidecode from django.template import defaultfilters slug = defaultfilters.slugify(unidecode(input_text)) 

Answer 2

The Mozilla website team has been working on an implementation : https://github.com/mozilla/unicode-slugify sample code at http://davedash.com/2011/03/24/how-we-slug-at-mozilla/