How to decide the language from cookies/headers/session in webapp2?

Question

I'd like to take advantage of webapp2's new features for localization that also has locale-specific formatting for time and currency.

Django has a good function called get_language_from_request that I made use of before I completely migrated to webapp2 and I now use the i18n from webapp2 instead and I can switch between localizations that I write with gettext and compile to files named messages.mo that my app can read and display. I've then identified and prioritized the following ways to get user's language: 1. HTTP GET eg. hl=pt-br for Brazilian Portuguese 2. HTTP SESSION variable I call i18n_language 3. Cookie I should set and get but I don't know exactly how 4. HTTP Header I could get and here I don't know exactly either and I'm looking how djnango does it with a convenient get_language_from_request that I used to use and now I've quit importing django and I still want this functionality for my now webapp2-based code.

def get_language_from_request(self, request):
    """
    Analyzes the request to find what language the user wants the system to
    show. If the user requests a sublanguage where we have a main language, we send
    out the main language.
    """
    if self.request.get('hl'):
      self.session['i18n_language'] = self.request.get('hl')
      return self.request.get('hl')

    if self.session:
      lang_code = self.session.get('i18n_language', None)
      if lang_code:
        logging.info('language found in session')
        return lang_code

    lang_code = Cookies(self).get(LANGUAGE_COOKIE_NAME)
    if lang_code:
        logging.info('language found in cookies')
        return lang_code

    accept = os.environ.get('HTTP_ACCEPT_LANGUAGE', '')
    for accept_lang, unused in self.parse_accept_lang_header(accept):
      logging.info('accept_lang:'+accept_lang)
      lang_code = accept_lang

    return lang_code

I see the django code is available but I don't know how much the i18n from webapp2 does for instance do I have to take care of fallback for languages such as pt-br should fall back to pt if there is no .mo localization for pt-br and similar for other dialects.

Actually setting the language I can do with

i18n.get_i18n().set_locale(language)

I ask for your help to make the priority for the different ways to get user language and I would also like to know your ideas how to go on with the implementation. Or do you think that I can do with just using session variable and not be this thorough about a "complete" solution since I anyway mostly fix the language for a geographical usage where my only actual used translations now are Brazilian Portuguese and English but I want it well prepared to switch to Spanish and Russian and other languages also, therefore I'd like to be able to switch to user language and at least save it to the webapp2 session and know what you think about using also cookie and header to get user language.

The original code I used to have si from django and looks like this and I can't use it anymore because it's locked to django.mo files and specific for django

def get_language_from_request(request):
    """
    Analyzes the request to find what language the user wants the system to
    show. Only languages listed in settings.LANGUAGES are taken into account.
    If the user requests a sublanguage where we have a main language, we send
    out the main language.
    """
    global _accepted
    from django.conf import settings
    globalpath = os.path.join(os.path.dirname(sys.modules[settings.__module__].__file__), 'locale')
    supported = dict(settings.LANGUAGES)

    if hasattr(request, 'session'):
        lang_code = request.session.get('django_language', None)
        if lang_code in supported and lang_code is not None and check_for_language(lang_code):
            return lang_code

    lang_code = request.COOKIES.get(settings.LANGUAGE_COOKIE_NAME)

    if lang_code and lang_code not in supported:
        lang_code = lang_code.split('-')[0] # e.g. if fr-ca is not supported fallback to fr

    if lang_code and lang_code in supported and check_for_language(lang_code):
        return lang_code

    accept = request.META.get('HTTP_ACCEPT_LANGUAGE', '')
    for accept_lang, unused in parse_accept_lang_header(accept):
        if accept_lang == '*':
            break

        # We have a very restricted form for our language files (no encoding
        # specifier, since they all must be UTF-8 and only one possible
        # language each time. So we avoid the overhead of gettext.find() and
        # work out the MO file manually.

        # 'normalized' is the root name of the locale in POSIX format (which is
        # the format used for the directories holding the MO files).
        normalized = locale.locale_alias.get(to_locale(accept_lang, True))
        if not normalized:
            continue
        # Remove the default encoding from locale_alias.
        normalized = normalized.split('.')[0]

        if normalized in _accepted:
            # We've seen this locale before and have an MO file for it, so no
            # need to check again.
            return _accepted[normalized]

        for lang, dirname in ((accept_lang, normalized),
                (accept_lang.split('-')[0], normalized.split('_')[0])):
            if lang.lower() not in supported:
                continue
            langfile = os.path.join(globalpath, dirname, 'LC_MESSAGES',
                    'django.mo')
            if os.path.exists(langfile):
                _accepted[normalized] = lang
                return lang

    return settings.LANGUAGE_CODE

Is it OK to do this for every request? And I think I should also set the header to the language self.response.headers['Content-Language'] = language

According to my expectation I can take some function directly from django if I choose to use the http headers but I don't understand what it does so maybe you can explain this code for me from django:

def parse_accept_lang_header(lang_string):
    """
    Parses the lang_string, which is the body of an HTTP Accept-Language
    header, and returns a list of (lang, q-value), ordered by 'q' values.

    Any format errors in lang_string results in an empty list being returned.
    """
    result = []
    pieces = accept_language_re.split(lang_string)
    if pieces[-1]:
        return []
    for i in range(0, len(pieces) - 1, 3):
        first, lang, priority = pieces[i : i + 3]
        if first:
            return []
        priority = priority and float(priority) or 1.0
        result.append((lang, priority))
    result.sort(lambda x, y: -cmp(x[1], y[1]))
    return result

Thank you

Update

I found that I couldn't use sessions in the initialize function of the request handler, maybe it's because the session object is not yet created. So I put the code for getting the language from the session i the BaseHandler render function and it appears to work. It would also be nice to consider the headers or cookie value.

Answer 1

Here's what I do - I have a base request handler that all my request handlers inherit from, then in here I have a constant that contains the available languages, and I override the init method to set the language on each request:

import webapp2
from webapp2_extras import i18n

AVAILABLE_LOCALES = ['en_GB', 'es_ES']

class BaseHandler(webapp2.RequestHandler):
    def __init__(self, request, response):
        """ Override the initialiser in order to set the language.
        """
        self.initialize(request, response)

        # first, try and set locale from cookie
        locale = request.cookies.get('locale')
        if locale in AVAILABLE_LOCALES:
            i18n.get_i18n().set_locale(locale)
        else:
            # if that failed, try and set locale from accept language header
            header = request.headers.get('Accept-Language', '')  # e.g. en-gb,en;q=0.8,es-es;q=0.5,eu;q=0.3
            locales = [locale.split(';')[0] for locale in header.split(',')]
            for locale in locales:
                if locale in AVAILABLE_LOCALES:
                    i18n.get_i18n().set_locale(locale)
                    break
            else:
                # if still no locale set, use the first available one
                i18n.get_i18n().set_locale(AVAILABLE_LOCALES[0])

First I check the cookie, then the header, finally defaulting to the first available language if a valid one wasn't found.

To set the cookie, I have a separate controller that looks something like this:

import base

class Index(base.BaseHandler):
    """ Set the language cookie (if locale is valid), then redirect back to referrer
    """
    def get(self, locale):
        if locale in self.available_locales:
            self.response.set_cookie('locale', locale, max_age = 15724800)  # 26 weeks' worth of seconds

        # redirect to referrer or root
        url = self.request.headers.get('Referer', '/')
        self.redirect(url)

So a URL like www.example.com/locale/en_GB would change the locale to en_GB, setting the cookie and returning to the referrer (this has the advantage of being able to switch languages on any page, and have it stay on the same page).

This method does not take into account partial matches for locales in the header, for instance "en" instead of "en_GB", but seeing as the list of languages I have enabled in the app is fixed (and the locale change URLs are hard-coded in the footer), I'm not too worried about it.

HTH

Answer 2

Totally based on fishwebby's answer and with some improvements and some design changes, here's what I do:

"""
Use this handler instead of webapp2.RequestHandler to support localization.
Fill the AVAILABLE_LOCALES tuple with the acceptable locales.
"""


__author__ = 'Cristian Perez <http://cpr.name>'


import webapp2
from webapp2_extras import i18n


AVAILABLE_LOCALES = ('en_US', 'es_ES', 'en', 'es')


class LocalizedHandler(webapp2.RequestHandler):

    def set_locale_from_param(self):
        locale = self.request.get('locale')
        if locale in AVAILABLE_LOCALES:
            i18n.get_i18n().set_locale(locale)
            # Save locale to cookie for future use
            self.save_locale_to_cookie(locale)
            return True
        return False

    def set_locale_from_cookie(self):
        locale = self.request.cookies.get('locale')
        if locale in AVAILABLE_LOCALES:
            i18n.get_i18n().set_locale(locale)
            return True
        return False

    def set_locale_from_header(self):
        locale_header = self.request.headers.get('Accept-Language')  # e.g. 'es,en-US;q=0.8,en;q=0.6'
        if locale_header:
            locale_header = locale_header.replace(' ', '')
            # Extract all locales and their preference (q)
            locales = []  # e.g. [('es', 1.0), ('en-US', 0.8), ('en', 0.6)]
            for locale_str in locale_header.split(','):
                locale_parts = locale_str.split(';q=')
                locale = locale_parts[0]
                if len(locale_parts) > 1:
                    locale_q = float(locale_parts[1])
                else:
                    locale_q = 1.0
                locales.append((locale, locale_q))

            # Sort locales according to preference
            locales.sort(key=lambda locale_tuple: locale_tuple[1], reverse=True)
            # Find first exact match
            for locale in locales:
                for available_locale in AVAILABLE_LOCALES:
                    if locale[0].replace('-', '_').lower() == available_locale.lower():
                        i18n.get_i18n().set_locale(available_locale)
                        return True

            # Find first language match (prefix e.g. 'en' for 'en-GB')
            for locale in locales:
                for available_locale in AVAILABLE_LOCALES:
                    if locale[0].split('-')[0].lower() == available_locale.lower():
                        i18n.get_i18n().set_locale(available_locale)
                        return True

        # There was no match
        return False

    def set_locale_default(self):
        i18n.get_i18n().set_locale(AVAILABLE_LOCALES[0])

    def save_locale_to_cookie(self, locale):
        self.response.set_cookie('locale', locale)

    def __init__(self, request, response):
        """
        Override __init__ in order to set the locale
        Based on: http://stackoverflow.com/a/8522855/423171
        """

        # Must call self.initialze when overriding __init__
        # http://webapp-improved.appspot.com/guide/handlers.html#overriding-init
        self.initialize(request, response)

        # First, try to set locale from GET parameter (will save it to cookie)
        if not self.set_locale_from_param():
            # Second, try to set locale from cookie
            if not self.set_locale_from_cookie():
                # Third, try to set locale from Accept-Language header
                if not self.set_locale_from_header():
                    # Fourth, set locale to first available option
                    self.set_locale_default()

1. It checks for the locale parameter in the URL, and if it exits, it sets a cookie with that locale for future use. In that way you can change the locale anywhere just using that locale parameter, but still avoid the parameter in upcoming requests.

2. If there is no parameter, it checks for the locale cookie.

3. If there is no cookie, it checks for the Accept-Language header. Very importantly, it takes into account the q preference factor of the header and also performs some little magic: language prefixes are accepted. For example, if the browser specifies en-GB but it doesn't exist in the AVAILABLE_LOCALES tuple, en will be selected if it exists, which will work by default with en_US if the locales for en do not exist. It also takes care of casing and format (- or _ as separator).