How can using gettext help me here?

Question

I am trying to set up a way to allow members to translate strings into other languages. You can see an example here: TRANSLATIONS TEST

Someone recommended that I use php's native gettext() function for this, instead of what I am already using to load the language files, which is this:

function loadLanguageFile($language, $file) {

    $temp = array();

    $data = file_get_contents('./'.$language.'/'.$file.'.'.$language.'.php');

    $codes = array (
        '/(\'\s*\.\s*\$)(.+?)(\s*\.\s*\')/',
        '/(=\s*\$)(.+?)(\s*\.\s*\')/',
        '/(\'\s*\.\s*\$)(.+?)(;)/',
        '/(\[\')(.+?)(\'\])/',
        '/<\?php/s', '/\?>/s', '/<\?/s'
    );
    $html = array (
        '{$2}',
        '= \'{$2}',
        '{$2}\';',
        '[$2]',
        '',
    );
    // Since we don't have the values for the vars.
    $data = preg_replace($codes, $html, $data);

    // We must change this because they are global.
    $data = str_replace('$txt', '$langEditor_txt', $data);
    $data = str_replace('$helptxt', '$langEditor_helptxt', $data);

    eval($data);

    if (isset($langEditor_txt)) {
        $temp['txt'] = $langEditor_txt;
        unset($GLOBALS['langEditor_txt']);
    }
    if (isset($langEditor_helptxt)) {
        $temp['helptxt'] = $langEditor_helptxt;
        unset($GLOBALS['langEditor_helptxt']);
    }

    return $temp;
}

The strings are contained within a file that is named like so: ManageDPModules.english.php DreamPortal.english.php etc.

These files can look like the following, when opened in any php editor, and can have many of these $txt variables:

<?php 
// Dream Portal (c) 2009-2010 Dream Portal Team 
// DreamPortal.english.php; @1.1 

global $scripturl, $context; 

// General Strings 
$txt['forum'] = 'Forum'; 
$txt['dream_portal'] = 'Dream Portal'; 
$txt['dp_core_modules'] = 'Collapse or Expand this Module'; 
$txt['dp_who_forum'] = 'Viewing the forum index of <a href="' . $scripturl . '?action=forum">' . $context['forum_name'] . '</a>.'; 
$txt['dp_who_portal'] = 'Viewing the portal index of <a href="' . $scripturl . '">' . $context['forum_name'] . '</a>.'; 
$txt['dp_who_page'] = 'Viewing the page &quot;<a href="' . $scripturl . '?page=%1$s">%2$s</a>&quot;.'; 
?>

I am using the following function to save the translations:

function langSave($lang, $file) {

    // We just don't get values from the form, they have to exist in the english files to be taken seriously.
    $default = loadLanguageFile('english', $file);

    if ($default['txt']) {
        foreach ($default['txt'] as $key=>$string) {
            if (isset($_REQUEST['txt'.$key]) && str_replace(' ', '', $_REQUEST['txt'.$key]) != '') {
                $data.='$txt[\''.$key.'\'] = \''.str_replace("'", "\'", $_REQUEST['txt'.$key]).'\';'."\n";
            }
        }
    }
    if ($default['helptxt']) {
        foreach ($default['helptxt'] as $key=>$string) {
            if (isset($_REQUEST['helptxt'.$key]) && str_replace(' ', '', $_REQUEST['helptxt'.$key]) != '') {
                $data.='$helptxt[\''.$key.'\'] = \''.str_replace("'", "\'", $_REQUEST['helptxt'.$key]).'\';'."\n";
            }
        }
    }

    if (isset($data)) {
        $codes = array (// '' . $test . '
            '/(\{)(.+?)(\})/',
            '/(\'\' \. \$)(.+?)( \. \')/',
            '/(\' \. \$)(.+?)( \. \'\')/',
            '/(\[\')(.+?)(\'\])/',
            '/(\[)(.+?)(\])/',
        );
        $html = array (
            '\' . \$$2 . \'',
            '\$$2 . \'',
            '\' . \$$2',
            '[$2]',
            '[\'$2\']',
        );
        // Convert the data back to normal.
        $data = preg_replace($codes, $html, $data);

        $data = '<?php'."\n".$data.'?>';
        file_put_contents('./'.$lang.'/'.$file.'.'.$lang.'.php', $data);
    }
    languageHome();
} 

Language function:

function languageHome() {

    $languages = loadLanguageList();

    echo '
    Language List
        <table>';
    foreach ($languages as $language) {
        echo '
            <tr>
                <td>
                    '.$language.'
                </td>
                <td>
                    <a href="index.php?op=langView&lang='.$language.'">View</a>
                </td>
            </tr>';
    }
    echo '
        </table>';

}

I fail to see how gettext will help out. There is no way to update the text catalog without rebooting the server every time. Maybe if someone can create a demo of this for me?

Also, would like it to support UTF-8. The data should be consistent.

So what is wrong with this implementation?? Why use gettext?? How can it be used to improve translations to work for both UTF-8 and non UTF-8 language strings so that it can be translated.?

EDIT: Please note, the files will eventually need to be renamed to: ManageDPModules.[language].php, DreamPortal.[language].php, etc., etc. in order for the translations to work. So, how would catalogs help me in this regard? If you want to see possible END-RESULT Translations, you can download a language package located here and open up the .german.php language files to see what it should look like after the member submits a language on a file by file basis. Noted that some of these packages have UTF-8 strings, and some do not. The filename of the packages let you know this. Would be nice if I can also make it support UTF-8, but it is not a requirement. Please note, I'm not trying to create complete packages here. I just want to create the languagefile.[language].php with all of the translated strings inside of them (which my code already does).

OK, I will provide the ENTIRE index.php file for this so you can see what it does exactly when you do translations. Here is the index.php file for this and you'll need some english language files: DreamPortal.english.php, ManageDPModules.english.php, and DreamHelp.english-utf8.php. Now, in order to see this, you need to upload to a server, index.php, create a few folders where index.php is, call 1 english, and create a folder in there for each additional language you want (I did 2 folders, spanish and french), than upload the 3 language files into the english folder. Run index.php in your browser and you will see it working.

Now, how could I use gettext for catalogs with this SAME approach. I need to enable online translation of files. I need to create PHP files of the translations in the SAME style that the .english.php files are with the same PREFIX that is before .english.php, and I need to change the language within the filename to the same language defined for the folder name. Online translations is the ONLY method available. Translator's need to focus ONLY on translating the strings. They shouldn't be focusing on installing programs, packaging it up, renaming the files, etc. etc.. This makes this process as painless as possible allowing it to be done online. And I know there is a way to do this, and even for UTF-8 support. But I'm using the best method that I know how at the moment. But so many of you are MUCH smarter at this sort of thing than I am, so I ask for help from you guys.

Is there anyone that can show me a better way? An example like the one that I have given you in this question?

I need to allow translations to be done ONLINE from translators, and would also like it to support UTF-8 files, as well as non UTF-8 files. I like the way I have it setup for the link provided above (TRANSLATIONS TEST) so that translator's can just do the translations online and not have to worry about anything else, and it would automatically create the files needed. Which would be the same as the english filename of the language with the folder name (representing the language) after the first . (dot) and it needs to have the extension .php (like it does in the code I am currently using). So basically, I need an adaption of the current index.php to support UTF-8 and non UTF-8 for all or most languages, and was told that using gettext() and catalog files would help with this.

Looking for a modified version of my current index.php to use gettext() in a way that it will support most, if not all, languages and translations. The REGEX I got going on for preg_replace isn't completely satisfactory because it seems to place a forward slash in front of double quotes when saving/submitting the translations. So perhaps an improvement on the preg_replace would be needed also.

I provided a complete example with ACTUAL CODE bytheway. I'd like for someone to alter this example, with the CODE that I provided to USE GETTEXT instead and support UTF-8. Or actually provide a ACTUAL METHOD for me to do this myself with. Not looking for a bunch of links that I can find on my own!

Thank You!

Answer 1

Why to re-invent the wheel? Maybe I don't understand but I'm pretty sure that what you're trying to do has been done before.

Just use one of the many-many existing solutions.

• Visit each of them & filter out the irrelevant projects
• Learn the differences between the rest and choose the one that suits you the best
• Continue develop & customize your own needs based on the existing project
• If you've found a bug or if you think the original project will benefits from your additions contact the author nd inform him

Here are some things to think about, when making your decisions:

• The language the project uses
• The ability to extend, add features and customize to your own needs
• Does it a service? or an open source you can download & use at your own server
• Does it support translation of plurals?
• The community & resources available for this project
• Does it also comes with a web-based UI like you've requested?

Here are some (randomly ordered):

• entrans
• pootle
• glotpress
• rosetta
• globalsight
• tcktranslator
• kartouche
• subtitrans
• tmsphp
• i18nedit
• Open Translation Engine
• GNU GetText Translator
• SiteTranslator
• phptrans
• myGengo's String
• NetBabel
• translatewiki

Good luck!

Update: thanks, El Yobo. I've marked this as a community wiki, anyone is welcome to edit or reply with another projects if he find something else.

Answer 2

When you have been recommended to use gettext, it was actually an advise to use a gettext-like translation system. Your current code is complex because of mnemonic text indicies. And the trouble you got into with the regular expressions for editing is caused by fiddling with variables in there. Let me propose some alternative code for transitional purposes.

The beauty of gettext lies in its use of a non-obtrusive API. The function call _() is simple enough to be thoroughly used without adding much syntax or code bloat. It's preferrable to things like getTextTrans('ABBR_TXT_ID'). Using such mnemonic text ids is a widespread fallacy; because in practice there aren't frequent rewordings and _("Raw english original text.") serves the same purpose. However, since you already have mnemonic keys in place, keep them if it's too much to change. This is just a recommendation.

Your real problem is the use of inline PHP expressions to build up the translation target strings. They are why the regular expressions for your translation editor became opaque. Therefore I'd highly recommend to use static strings and provide placeholders. The translation function should be tasked with handling it. (Don't worry about microoptimizing here!) - I would use {$url_xy} PHP/Smarty-style placeholders for example:

$txt['dp_who_forum'] = 'Viewing the forum index
of <a href="{$scripturl}?action=forum">{$forum_name}</a>.'; 

And a translation function that looks up a global placeholder table or params ($context) for replacing:

function __($text, $params=array()) {
    global $txt, $txt_placeholders;
    if (isset($txt[$text])) {
        $text = $txt[$text];
    }
    if (strpos($text, '{$')) {
        $params = array_merge($params, $txt_placeholders);
        $text= preg_replace("/\{\$(\w+)\}/e", "$params['$1']", $text);
    }
    return $text;
}

Optimizable. But this way you can use a static mnemonic->text or english->text set of translation arrays. You only use static strings in your text-translation-editor-form thingy. Those static strings are shown as-is, and your translators edit the english text, but not any of the {$placeholders}.

Hence your code for the translation feature won't need any of the complex regular expressions (in this case they are not useful) to match strings and inline PHP variables. In fact a much simpler include() and var_export() combination can now take its place:

function langSave($lang, $file) {
    $txt = array();
    include($file);   // assuming it contains a simple $txt = array(...

    foreach ($txt as $key=>$string) {
        $txt[$key] = $_REQUEST["txt$key"];
    }
    file_put_contents($file, "<?php\n\$txt =".var_export($txt,1).';?>');
}

Filehandling and whatnot needs to be customized of course. But still this is a simpler approach.

And it would further allow you to transition to one of the gettext variations as backends. Keep your custom wrapper function __() and use e.g. Zend_Translate as backend. This allows you to keep using your .php $txt=array() translation files (or I think so) or move to gettext-style .mo/.po files. The advantage of that being, that there is rich tool support in contrast to homebrew solutions.

Anyway this is how you use the stuff in your main application:

print "<a href='...'>" . __("link_text") . "</a>";
print __("dp_forum_link");   // uses e.g. $txt_placeholder["scripturl"]
print __("dp_param_link", array("scripturl"=>"http://override..."));

Moving from the short_txt keys to english->foreign text translation arrays would be advisable first, but in theory any of the backends from dvbs answer would be applicable. Now you already said that native gettext is not an option for you (for non-fastcgi PHP setups the memory resistance is a drawback). But PHPs native INTL features might be of help to you. In particular http://www.php.net/manual/en/class.messageformatter.php might be more useful than the simple-minded {$var} replacement wrapper I gave you. But I've never used it, and I think Zend_Translate is likely more useful. In particular can any of those backends give you charset independence. Personally I'd just stick to UTF-8, no matter what. But gettext .mo/.po files can each have their own custom charset for example.