Best practice multi language website

Question

I've been struggling with this question for quite some months now, but I haven't been in a situation that I needed to explore all possible options before. Right now, I feel like it's time to get to know the possibilities and create my own personal preference to use in my upcoming projects.

Let me first sketch the situation I'm looking for

I'm about to upgrade/redevelop a content management system which I've been using for quite a while now. However, I'm feeling multi language is a great improvement to this system. Before I did not use any frameworks but I'm going to use Laraval4 for the upcoming project. Laravel seems the best choice of a cleaner way to code PHP. Sidenote: Laraval4 should be no factor in your answer. I'm looking for general ways of translation that are platform/framework independent.

What should be translated

As the system I am looking for needs to be as user friendly as possible the method of managing the translation should be inside the CMS. There should be no need to start up an FTP connection to modify translation files or any html/php parsed templates.

Furthermore, I'm looking for the easiest way to translate multiple database tables perhaps without the need of making additional tables.

What did I come up with myself

As I've been searching, reading and trying things myself already. There are a couple of options I have. But I still don't feel like I've reached a best practice method for what I am really seeking. Right now, this is what I've come up with, but this method also has it side effects.

1. PHP Parsed Templates: the template system should be parsed by PHP. This way I'm able to insert the translated parameters into the HTML without having to open the templates and modify them. Besides that, PHP parsed templates gives me the ability to have 1 template for the complete website instead of having a subfolder for each language (which I've had before). The method to reach this target can be either Smarty, TemplatePower, Laravel's Blade or any other template parser. As I said this should be independent to the written solution.
2. Database Driven: perhaps I don't need to mention this again. But the solution should be database driven. The CMS is aimed to be object oriented and MVC, so I would need to think of a logical data structure for the strings. As my templates would be structured: templates/Controller/View.php perhaps this structure would make the most sense: Controller.View.parameter. The database table would have these fields a long with a value field. Inside the templates we could use some sort method like echo __('Controller.View.welcome', array('name', 'Joshua')) and the parameter contains Welcome, :name. Thus the result being Welcome, Joshua. This seems a good way to do this, because the parameters such as :name are easy to understand by the editor.
3. Low Database Load: Of course the above system would cause loads of database load if these strings are being loaded on the go. Therefore I would need a caching system that re-renders the language files as soon as they are edited/saved in the administration environment. Because files are generated, also a good file system layout is needed. I guess we can go with languages/en_EN/Controller/View.php or .ini, whatever suits you best. Perhaps an .ini is even parsed quicker in the end. This fould should contain the data in the format parameter=value; . I guess this is the best way of doing this, since each View that is rendered can include it's own language file if it exists. Language parameters then should be loaded to a specific view and not in a global scope to prevent parameters from overwriting each other.
4. Database Table translation: this in fact is the thing I'm most worried about. I'm looking for a way to create translations of News/Pages/etc. as quickly as possible. Having two tables for each module (for example News and News_translations) is an option but it feels like to much work to get a good system. One of the things I came up with is based on a data versioning system I wrote: there is one database table name Translations, this table has a unique combination of language, tablename and primarykey. For instance: en_En / News / 1 (Referring to the English version of the News item with ID=1). But there are 2 huge disadvantages to this method: first of all this table tends to get pretty long with a lot of data in the database and secondly it would be a hell of a job to use this setup to search the table. E.g. searching for the SEO slug of the item would be a full text search, which is pretty dumb. But on the other hand: it's a quick way to create translatable content in every table very fast, but I don't believe this pro overweights the con's.
5. Front-end Work: Also the front-end would need some thinking. Of course we would store the available languages in a database and (de)active the ones we need. This way the script can generate a dropdown to select a language and the back-end can decide automatically what translations can be made using the CMS. The chosen language (e.g. en_EN) would then be used when getting the language file for a view or to get the right translation for a content item on the website.

So, there they are. My ideas so far. They don't even include localization options for dates etc yet, but as my server supports PHP5.3.2+ the best option is to use the intl extension as explained here: http://devzone.zend.com/1500/internationalization-in-php-53/ - but this would be of use in any later stadium of development. For now the main issue is how to have the best practics of translation of the content in a website.

Besides everything I explained here, I still have another thing which I haven't decided yet, it looks like a simple question, but in fact it's been giving me headaches:

URL Translation? Should we do this or not? and in what way?

So.. if I have this url: http://www.domain.com/about-us and English is my default language. Should this URL be translated into http://www.domain.com/over-ons when I choose Dutch as my language? Or should we go the easy road and simply change the content of the page visible at /about. The last thing doesn't seem a valid option because that would generate multiple versions of the same URL, this indexing the content will fail the right way.

Another option is using http://www.domain.com/nl/about-us instead. This generates at least a unique URL for each content. Also this would be easier to go to another language, for example http://www.domain.com/en/about-us and the URL provided is easier to understand for both Google and Human visitors. Using this option, what do we do with the default languages? Should the default language remove the language selected by default? So redirecting http://www.domain.com/en/about-us to http://www.domain.com/about-us ... In my eyes this is the best solution, because when the CMS is setup for only one language there is no need to have this language identification in the URL.

And a third option is a combination from both options: using the "language-identification-less"-URL (http://www.domain.com/about-us) for the main language. And use an URL with a translated SEO slug for sublanguages: http://www.domain.com/nl/over-ons & http://www.domain.com/de/uber-uns

I hope my question gets your heads cracking, they cracked mine for sure! It did help me already to work things out as a question here. Gave me a possibility to review the methods I've used before and the idea's I'm having for my upcoming CMS.

I would like to thank you already for taking the time to read this bunch of text!

// Edit #1:

I forgot to mention: the __() function is an alias to translate a given string. Within this method there obviously should be some sort of fallback method where the default text is loaded when there are not translations available yet. If the translation is missing it should either be inserted or the translation file should be regenerated.

Answer 1

Topic's premise

There are three distinct aspects in a multilingual site:

• interface translation
• content
• url routing

While they all interconnected in different ways, from CMS point of view they are managed using different UI elements and stored differently. You seem to be confident in your implementation and understanding of the first two. The question was about the latter aspect - "URL Translation? Should we do this or not? and in what way?"

What the URL can be made of?

A very important thing is, don't get fancy with IDN. Instead favor transliteration (also: transcription and romanization). While at first glance IDN seems viable option for international URLs, it actually does not work as advertised for two reasons:

• some browsers will turn the non-ASCII chars like 'ч' or 'ž' into '%D1%87' and '%C5%BE'
• if user has custom themes, the theme's font is very likely to not have symbols for those letters

I actually tried to IDN approach few years ago in a Yii based project (horrible framework, IMHO). I encountered both of the above mentioned problems before scraping that solution. Also, I suspect that it might be an attack vector.

Available options ... as I see them.

Basically you have two choices, that could be abstracted as:

• http://site.tld/[:query]: where [:query] determines both language and content choice

• http://site.tld/[:language]/[:query]: where [:language] part of URL defines the choice of language and [:query] is used only to identify the content

Query is Α and Ω ..

Lets say you pick http://site.tld/[:query].

In that case you have one primary source of language: the content of [:query] segment; and two additional sources:

• value $_COOKIE['lang'] for that particular browser
• list of languages in HTTP Accept-Language ^{(1), (2)} header

First, you need to match the query to one of defined routing patterns (if your pick is Laravel, then read here). On successful match of pattern you then need to find the language.

You would have to go through all the segments of the pattern. Find the potential translations for all of those segments and determine which language was used. The two additional sources (cookie and header) would be used to resolve routing conflicts, when (not "if") they arise.

Take for example: http://site.tld/blog/novinka.

That's transliteration of "блог, новинка", that in English means approximately "blog", "latest".

As you can already notice, in Russian "блог" will be transliterated as "blog". Which means that for the first part of [:query] you (in the best case scenario) will end up with ['en', 'ru'] list of possible languages. Then you take next segment - "novinka". That might have only one language on the list of possibilities: ['ru'].

When the list has one item, you have successfully found the language.

But if you end up with 2 (example: Russian and Ukrainian) or more possibilities .. or 0 possibilities, as a case might be. You will have to use cookie and/or header to find the correct option.

And if all else fails, you pick the site's default language.

Language as parameter

The alternative is to use URL, that can be defined as http://site.tld/[:language]/[:query]. In this case, when translating query, you do not need to guess the language, because at that point you already know which to use.

There is also a secondary source of language: the cookie value. But here there is no point in messing with Accept-Language header, because you are not dealing with unknown amount of possible languages in case of "cold start" (when user first time opens site with custom query).

Instead you have 3 simple, prioritized options:

1. if [:language] segment is set, use it
2. if $_COOKIE['lang'] is set, use it
3. use default language

When you have the language, you simply attempt to translate the query, and if translation fails, use the "default value" for that particular segment (based on routing results).

Isn't here a third option?

Yes, technically you can combine both approaches, but that would complicate the process and only accommodate people who want to manually change URL of http://site.tld/en/news to http://site.tld/de/news and expect the news page to change to German.

But even this case could probable be mitigated using cookie value (which would contain information about previous choice of language), to implement with less magic and hope.

Which approach to use?

As you might already guessed, I would recommend http://site.tld/[:language]/[:query] as the more sensible option.

Also in real word situation you would have 3rd major part in URL: "title". As in name of the product in online shop or headline of article in news site.

Example: http://site.tld/en/news/article/121415/EU-as-global-reserve-currency

In this case '/news/article/121415' would be the query, and the 'EU-as-global-reserve-currency' is title. Purely for SEO purposes.

Can it be done in Laravel?

Kinda, but not by default.

I am not too familiar with it, but from what I have seen, Laravel uses simple pattern-based routing mechanism. To implement multilingual URLs you will probably have to extend core class(es), because multilingual routing need access to different forms of storage (database, cache and/or configuration files).

It's routed. What now?

As a result of all you would end up with two valuable pieces of information: current language and translated segments of query. These values then can be used to dispatch to the class(es) which will produce the result.

Basically, the following URL: http://site.tld/ru/blog/novinka (or the version without '/ru') gets turned into something like

$parameters = [    'language' => 'ru',    'classname' => 'blog',    'method' => 'latest', ]; 

Which you just use for dispatching:

$instance = new {$parameter['classname']}; $instance->{'get'.$parameters['method']}( $parameters ); 

.. or some variation of it, depending on the particular implementation.