json_encode() in PHP returns Chinese characters in escaped Unicode

Question

I’ve got a simple PHP one-dimension array.

When I do a var dump (echo var_dump($a)), I get this as the output:

array(3) { [0]=>  string(3) "尽" [1]=>  string(21) "exhausted||to exhaust" [2]=>  string(4) "jin3" }

However, when I json_encode it (echo json_encode($a)) I get this:

["\u5c3d","exhausted||to exhaust","jin3"]

The hex value that it’s returning is the correct one, but I can’t figure out how to stop it from giving me the hex. I just want it to display the character.

If I echo mb_internal_encoding() it returns UTF-8, which is what I’ve set it to. I’ve been very careful in all my string manipulation to use the mb_ functions so none of the data gets messed up.

I know that I could write a modified json_encode function which would take care of the problem. But I want to know what’s going on here.

Answer 1

I know this question is older but thought I'd lend my working–in–China to_json and to_utf8 functions — which includes some nice formatting (JSON_PRETTY_PRINT) when in development vs minified production. (Adapt to your own env/system)

Simple

// Produces JSON with Chinese Characters fully un-encoded.
// NOT RFC4627 compliant
json_encode($data, JSON_UNESCAPED_UNICODE|JSON_UNESCAPED_SLASHES); 

to_json()

function to_json($data, $pretty=null, $inculde_security=false, $try_to_recover=true) {
  // @Note: json_encode() *REQUIRES* data to be in valid UTF8 format BEFORE
  //                    trying to json_encode   and since we are working with Chinese
  //                    characters, we need to make sure that we explicitly allow:
  //                    JSON_UNESCAPED_UNICODE|JSON_UNESCAPED_SLASHES
  //                    *Unless a mode is explicitly passed into the function
    $json_encoded = '{}';
    if ($pretty === null && is_env_prod()) { // @NOTE: Substitute with your own Production env check
        $json_encoded = json_encode( $data, JSON_UNESCAPED_UNICODE|JSON_UNESCAPED_SLASHES );
    } else if ($pretty === null && is_env_dev()){ // @NOTE: Substitute with your own Development env check
        $json_encoded = json_encode( $data, JSON_PRETTY_PRINT|JSON_UNESCAPED_UNICODE|JSON_UNESCAPED_SLASHES );
    } else {
        // PRODUCTION
        $json_encoded = json_encode( $data, $pretty );
    }



    // (1) Do not return an error if the inital data was empty
    // (2) Return an error if json_encode() failed
    if (json_last_error() > 0) {
        if (!!$data || !empty($data)) {
            if (!$json_encoded == false || empty($json_encoded) || $json_encoded == '{}') {
                $json_encoded = json_encode([
                    'status' => false,
                    'error' => [
                        'json_last_error' => json_last_error(),
                        'json_last_error_msg' => json_last_error_msg()
                    ]
                ]);
            } else if (!!$try_to_recover) {
                // there was data in $data so lets try to forensically recover a little? by removing $k => $v pairs that fail to be JSON encoded
                foreach (((array) $data) as $k => $v) {
                    if (!json_encode([$k => $v])) {
                        if (is_array($data)) {
                            unset($data[$k]);
                        } else if (is_object($data)) {
                            unset($data->{$k});
                        }
                    }
                }

                // if the data still is not empty, and there is a status set in the data
                //      then set it to false and add a error message/data
                //      ONLY for Array & Objects
                if (!empty($json_encoded) && count($json_encoded) < 1) {
                    if (!json_encode($data)) {
                        if (is_array($json_encoded)) {
                            $json_encoded['status'] = false;
                            $json_encoded['message'] = "json_encoding_error";
                            $json_encoded['error'] = [
                                'json_last_error' => json_last_error(),
                                'json_last_error_msg' => json_last_error_msg()
                            ];
                        } else if (is_object($json_encoded)) {
                            $json_encoded->status = false;
                            $json_encoded->message = "json_encoding_error";
                            $json_encoded->error = [
                                'json_last_error' => json_last_error(),
                                'json_last_error_msg' => json_last_error_msg()
                            ];
                        }
                    } else {
                      // We have removed the offending data
                      return to_json($data, $pretty, $include_security, $try_to_recover);
                    }
                }

                // we've cleaned out any data that was causing the problem, and included
                //      false to indicate this is a one-time recursion recovery.
                return $this->to_json($pretty, $include_security, false);
            }
        } else { } // don't do anything as the value is already false
    }

  return ( ($inculde_security) ? ")]}',\n" : '' ) . $json_encoded;
}

---

Another funciton that might be usful is my recursive to_utf8() functionality:

to_utf8()

// @NOTE: Common Chinese GBK encoding: to_utf8($data, 'GB2312')
function to_utf8($in, $source_encoding='HTML-ENTITIES') {
  if (is_string($in)) {
    return mb_convert_encoding(
      $in,
      $source_encoding,
      'UTF-8'
    );
  } else if (is_array($in) || is_object($in)) {

    array_walk_recursive($in, function(&$item, &$key) {
      $key = to_utf8($key);

      if (is_object($item) || is_array($item)) {
        $item = to_utf8($item);
      } else {
        if (!mb_detect_encoding($item, 'UTF-8', true)){
          $item = utf8_encode($item);
        }
      }
    });

    $ret_object = is_object($in);
    return ($ret_object) ? (object) $in : (array) $in;
  }

  return $in;
}

---

Validate RFC4627 (valid JSON)

$pcre_regex = '
  /
  (?(DEFINE)
     (?<number>   -? (?= [1-9]|0(?!\d) ) \d+ (\.\d+)? ([eE] [+-]? \d+)? )
     (?<boolean>   true | false | null )
     (?<string>    " ([^"\\\\]* | \\\\ ["\\\\bfnrt\/] | \\\\ u [0-9a-f]{4} )* " )
     (?<array>     \[  (?:  (?&json)  (?: , (?&json)  )*  )?  \s* \] )
     (?<pair>      \s* (?&string) \s* : (?&json)  )
     (?<object>    \{  (?:  (?&pair)  (?: , (?&pair)  )*  )?  \s* \} )
     (?<json>   \s* (?: (?&number) | (?&boolean) | (?&string) | (?&array) | (?&object) ) \s* )
  )
  \A (?&json) \Z
  /six
';

$matches = false;
preg_match($pcre_regex, trim($body), $matches);

var_dump('RFC4627 Verification (Regex) ', [
  'has_passed' => (count($matches) == 1) ? 'YES' : 'NO',
  'matches'    => $matches
]);

is_json()

// One Liner
is_string($json_string) && !preg_match('/[^,:{}\\[\\]0-9.\\-+Eaeflnr-u \\n\\r\\t]/', preg_replace('/"(\\.|[^"\\\\])*"/', '', $json_string));

// Alt Function — more consistant
function is_json($json_string) {
  if (!is_string($json_string) || is_numeric($json_string)) {
      return false;
  }

  $val = @json_decode($json_string);

  return ($val != null) && (json_last_error() === JSON_ERROR_NONE);

  // Inconsistant results, reverted to json_decode() + JSON_ERROR_NONE check
  // return is_string($json_string) && !preg_match('/[^,:{}\\[\\]0-9.\\-+Eaeflnr-u \\n\\r\\t]/', preg_replace('/"(\\.|[^"\\\\])*"/', '', $json_string));
}

is_utf8()

function is_utf8($str) {
  if (is_array($str)) {
    foreach ($str as $k=>$v) {
      if (is_string($v) && !is_utf8($v)) {
        return false;
      }
    }
  }

  return (is_string($str) && preg_match('//u', $str));
}

Answer 2

The behaviour of json_encode() is perfectly correct, but unnecessary. In PHP 5.4, it can be disabled with the JSON_UNESCAPED_UNICODE flag.