Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to deal with non UTF-8 encoded urls in express

We have a node js application which we have recently moved over from running on IIS 7 (via IIS node) to running on Linux (Elastic Beanstalk). Since we switched, we've been getting a lot of non UTF-8 urls being sent to our application (mainly from crawlers) such as:

Bj%F6rk which IIS was converting to Björk. This is now being passed to our application and our web framework (express) eventually calls down to

decodeURIComponent('Bj%F6rk'); URIError: URI malformed at decodeURIComponent (native) at repl:1:1 at REPLServer.self.eval (repl.js:110:21) at repl.js:249:20 at REPLServer.self.eval (repl.js:122:7) at Interface.<anonymous> (repl.js:239:12) at Interface.emit (events.js:95:17) at Interface._onLine (readline.js:203:10) at Interface._line (readline.js:532:8) at Interface._ttyWrite (readline.js:761:14)

Is there a recommended safe way we can perform the same conversion as IIS before sending the url string to express?

Bearing in mind

  1. We are receiving requests to these badly encoded URLS and
  2. There is a way to decode them using the deprecated unescape javascript function and
  3. The majority of the requests to these URLs are coming from Bing Bot and we want to minimise any adverse effect on our search rankings.

    • Should we really be doing this for all incoming URLs?
    • Are there any security or performance implications we should be concerned about?
    • Should we be concerned about unescape being removed in the near future?
    • Is there a better / safer way to solve this problem (Yes we did read that MDN article linked to above)
like image 987
Will Munn Avatar asked Sep 18 '15 13:09

Will Munn


People also ask

Can URLs contain utf8?

Actually they both amount to "no". Neither domains nor URLs can contain any non-ASCII characters. However, there exist ways to encode arbitrary characters as ASCII (percent encoding and punycode)...

Is URL encoding necessary?

Why do we need to encode? URLs can only have certain characters from the standard 128 character ASCII set. Reserved characters that do not belong to this set must be encoded. This means that we need to encode these characters when passing into a URL.

How do I encode a URL?

URL Encoding (Percent Encoding) URLs can only be sent over the Internet using the ASCII character-set. Since URLs often contain characters outside the ASCII set, the URL has to be converted into a valid ASCII format. URL encoding replaces unsafe ASCII characters with a "%" followed by two hexadecimal digits.

What is URL encoded in node JS?

urlencoded() function is a built-in middleware function in Express. It parses incoming requests with urlencoded payloads and is based on body-parser. Parameter: The options parameter contains various property like extended, inflate, limit, verify etc.


1 Answers

Should we really be doing this for all incoming URLs?

No, you shouldn't. The request being made uses non-UTF8 URI components. That shouldn't be your problem.

Are there any security or performance implications we should be concerned about?

The encoding of a URI component is not a security issue. Injection attempts via querystring or path params are. But that's another subject. In terms of performance, every middleware will make your responses take a bit longer. But I wouldn't even worry about that. If you want to decode the URI yourself, just do it. It'll only take a few milliseconds.

Should we be concerned about unescape being removed in the near future?

Actually you should. unescape is deprecated. If you still want to use it; just check if it exists first. i.e. 'unescape' in global. You can also use the built-in alternate: require('querystring').unescape() which won't produce the same result in every case but it won't throw a URIError. (Not recommended though).

To minimise any adverse effect on search rankings:

Determine which status code your express app returns in these cases. It could be 500 (INTERNAL SERVER ERROR) which will look bad and 404 (NOT FOUND) which will tell the crawler you don't have a result for the query (which may not be true).

In these cases, I suggest you override this by returning a client error such as 400 (BAD REQUEST) instead, since the origin of the problem is a malformed URI component being requested, which should be in UTF-8 but it's not. The crawler/bot should be concerned about that.

// middleware for responding with BAD REQUEST
app.use(function (err, req, res, next) {
    if (err instanceof URIError) {
        res.status(400).send();
    }
});

Above all, trying to return a result for a malformed URI has other side effects. First, you'll be allowing a bad request — can't be good :). Secondly, it'll mean you have a result for a bad URI which will get stored by crawlers/bots when they get a 200 OK response and it will get spread. Then you'll have to deal with more bad requests.

To conclude; don't decode via unescape. Express already tries to decode via what's proper: decodeURIComponent. If that fails, let it be.

like image 162
Onur Yıldırım Avatar answered Sep 30 '22 06:09

Onur Yıldırım