Invalid characters (non UTF-8) from node url Request but valid from browser?

Question

I am calling Google Place api web service with Request in Node js. The body of the request gives an error Invalid request. One of the input parameters contains a non-UTF-8 string. because I use Khmer characters in the url parameters (keyword parameter).

nearByUrl = 'https://maps.googleapis.com/maps/api/place/nearbysearch/json?location=11.55082623,104.93225202&radius=400&keyword=ស្ថានីយប្រេង&key=' + MY_KEY;
request({
    url: nearByUrl,
    json: true
  }, function (error, response, body) {
    console.log(JSON.stringify(body, null, 2));
})

However, I can get a valid JSON with results when calling the exact same URL with Khmer characters from Chrome browser.

Is this problem related to Request?

How can I fix this?

Answer 1

So if you enter URL to which you want to send request in Chrome and open development tools, you will see that raw URL to which request is sent looks similar to this:

https://maps.googleapis.com/maps/api/place/nearbysearch/json?location=11.55082623,104.93225202&radius=400&keyword=%E1%9E%9F%E1%9F%92%E1%9E%90%E1%9E%B6%E1%9E%93%E1%9E%B8%E1%9E%99%E1%9E%94%E1%9F%92%E1%9E%9A%E1%9F%81%E1%9E%84

Basically Chrome encoded all query parameters to ASCII, and when you enter directly parameters to URL, query parameters are not encoded. But if you send your parameters to request library through qs object, library will encoded them for you, and you will not have issue from the question.

var request = require("request")

var option = {
  uri: 'https://maps.googleapis.com/maps/api/place/nearbysearch/json',
  qs: {
    location: '11.55082623,104.93225202',
    radius: 1000,
    keyword: 'ស្ថានីយប្រេង',
    key: MY_KEY
  }
};

request(
    option, function (error, response, body) {
    console.log(JSON.stringify(body, null, 2));
})

You could use method that is build into js library like this, but I personally think that first method is much better solution:

nearByUrl = 'https://maps.googleapis.com/maps/api/place/nearbysearch/json?location=11.55082623,104.93225202&radius=400&keyword=' + encodeURIComponent(escape('ស្ថានីយប្រេង')) + '&key=' + MY_KEY;
request({
    url: nearByUrl,
    json: true
  }, function (error, response, body) {
    console.log(JSON.stringify(body, null, 2));
})

Why I think that first solution with qs param is better solution, because library is doing it for you and all parameters are encoded.

Better explanation of second method can be found here.

Hopefully this is solution to your issue :)