We have a node js application which we have recently moved over from running on IIS 7 (via IIS node) to running on Linux (Elastic Beanstalk). Since we switched, we've been getting a lot of non UTF-8 urls being sent to our application (mainly from crawlers) such as:
Bj%F6rk
which IIS was converting to Björk
. This is now being passed to our application and our web framework (express) eventually calls down to
decodeURIComponent('Bj%F6rk');
URIError: URI malformed
at decodeURIComponent (native)
at repl:1:1
at REPLServer.self.eval (repl.js:110:21)
at repl.js:249:20
at REPLServer.self.eval (repl.js:122:7)
at Interface.<anonymous> (repl.js:239:12)
at Interface.emit (events.js:95:17)
at Interface._onLine (readline.js:203:10)
at Interface._line (readline.js:532:8)
at Interface._ttyWrite (readline.js:761:14)
Is there a recommended safe way we can perform the same conversion as IIS before sending the url string to express?
Bearing in mind
unescape
javascript function andThe majority of the requests to these URLs are coming from Bing Bot and we want to minimise any adverse effect on our search rankings.
unescape
being removed in the near future?Actually they both amount to "no". Neither domains nor URLs can contain any non-ASCII characters. However, there exist ways to encode arbitrary characters as ASCII (percent encoding and punycode)...
Why do we need to encode? URLs can only have certain characters from the standard 128 character ASCII set. Reserved characters that do not belong to this set must be encoded. This means that we need to encode these characters when passing into a URL.
URL Encoding (Percent Encoding) URLs can only be sent over the Internet using the ASCII character-set. Since URLs often contain characters outside the ASCII set, the URL has to be converted into a valid ASCII format. URL encoding replaces unsafe ASCII characters with a "%" followed by two hexadecimal digits.
urlencoded() function is a built-in middleware function in Express. It parses incoming requests with urlencoded payloads and is based on body-parser. Parameter: The options parameter contains various property like extended, inflate, limit, verify etc.
Should we really be doing this for all incoming URLs?
No, you shouldn't. The request being made uses non-UTF8 URI components. That shouldn't be your problem.
Are there any security or performance implications we should be concerned about?
The encoding of a URI component is not a security issue. Injection attempts via querystring or path params are. But that's another subject. In terms of performance, every middleware will make your responses take a bit longer. But I wouldn't even worry about that. If you want to decode the URI yourself, just do it. It'll only take a few milliseconds.
Should we be concerned about unescape being removed in the near future?
Actually you should. unescape
is deprecated. If you still want to use it; just check if it exists first. i.e. 'unescape' in global
. You can also use the built-in alternate: require('querystring').unescape()
which won't produce the same result in every case but it won't throw a URIError
. (Not recommended though).
To minimise any adverse effect on search rankings:
Determine which status code your express app returns in these cases. It could be 500 (INTERNAL SERVER ERROR) which will look bad and 404 (NOT FOUND) which will tell the crawler you don't have a result for the query (which may not be true).
In these cases, I suggest you override this by returning a client error such as 400 (BAD REQUEST) instead, since the origin of the problem is a malformed URI component being requested, which should be in UTF-8 but it's not. The crawler/bot should be concerned about that.
// middleware for responding with BAD REQUEST
app.use(function (err, req, res, next) {
if (err instanceof URIError) {
res.status(400).send();
}
});
Above all, trying to return a result for a malformed URI has other side effects. First, you'll be allowing a bad request — can't be good :). Secondly, it'll mean you have a result for a bad URI which will get stored by crawlers/bots when they get a 200 OK response and it will get spread. Then you'll have to deal with more bad requests.
To conclude; don't decode via unescape
. Express already tries to decode via what's proper: decodeURIComponent
. If that fails, let it be.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With