How would you create a string of all UTF-8 characters?

Question

There are many ways to represent the +1 million UTF-8 characters. Take the latin capital "A" with macron (Ā). This is unicode code point U+0100, hex number 0xc4 0x80, decimal number 196 128, and binary 11000100 10000000.

I would like to create a collection of the first 65,535 UTF-8 characters for use in testing applications. These are all unicode characters up to code point U+FFFF (byte3).

Is it possible to do something like a for($x=0) loop and then convert the resulting decimal to another base (like hex) which would allow the creation of the matching unicode character?

I can create the value Ā using something like this:

$char = "\xc4\x80";
// or
$char = chr(196).chr(128);

However, I am not sure how to turn this into an automated process.

// fail!
$char = "\x". dechex($a). "\x". dexhex($b);

Answer 1

You can leverage iconv (or a few other functions) to convert a code point number to a UTF-8 string:

function unichr($i)
{
    return iconv('UCS-4LE', 'UTF-8', pack('V', $i));
}

$codeunits = array();
for ($i = 0; $i<0xD800; $i++)
    $codeunits[] = unichr($i);
for ($i = 0xE000; $i<0xFFFF; $i++)
    $codeunits[] = unichr($i);
$all = implode($codeunits);

(I avoided the surrogate range 0xD800–0xDFFF as they aren't valid to put in UTF-8 themselves; that would be “CESU-8”.)

Answer 2

I'm not sure you can do this programmatically, mostly because there is a difference between a Unicode code point and a character. See http://www.unicode.org/standard/where for a few examples of characters that are represented by a combination of code points.

Some code points make no sense on their own and can only be used in conjunction with another character (think accents). See http://www.unicode.org/charts/charindex.html for a list of code points, and look at the section with all the "combining" code points.

Also, for use in testing applications, you'd need something else besides a list of possible UTF-8 code points, namely several invalid/malformed UTF-8 sequences that your app needs to be able to recover gracefully from.

For this, take a look at Markus Kuhn's Unicode stress test.

Answer 3

I quickly translated this from C, but it should give you the idea:

function encodeUTF8( $inValue ) {
    $result = "";

    if ( $inValue < 0x00000080 ) {
        $result .= chr( $inValue );
        $extra = 0;
    } else if ( $inValue < 0x00000800 ) {
        $result .= chr( 0x00C0 | ( ( $inValue >> 6 ) & 0x001F ) );
        $extra = 6;
    } else if ( $inValue < 0x00010000 ) {
        $result .= chr( 0x00E0 | ( ( $inValue >> 12 ) & 0x000F ) );
        $extra = 12;
    } else if ( $inValue < 0x00200000 ) {
        $result .= chr( 0x00F0 | ( ( $inValue >> 18 ) & 0x0007 ) );
        $extra = 18;
    } else if ( $inValue < 0x04000000 ) {
        $result .= chr( 0x00F8 | ( ( $inValue >> 24 ) & 0x0003 ) );
        $extra = 24;
    } else if ( $inValue < 0x80000000 ) {
        $result .= chr( 0x00FC | ( ( $inValue >> 30 ) & 0x0001 ) );
        $extra = 30;
    }

    while ( $extra > 0 ) {
        $result .= chr( 0x0080 | ( ( $inValue >> ( $extra -= 6 ) ) & 0x003F ) );
    }

    return $result;
}

The logic is sound but I am not sure about the php so be sure to check it over. I have never tried to use chr like this.

There are a lot of values that you would not want to encode, like 0xD000-0xDFFF, 0xE000-0xF8FF and 0xFFF0-0xFFFF, and there are several other gaps for combining characters and reserved characters.