Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Node.js request turns umlauts to �

I'm doing the following request (using request/request) against a web service:

return request.postAsync({
    url,
    charset: 'Cp1252', // I also tried utf-8
    encoding: null, //
    // I also tried Cp1252 -> unknown encoding,
    // I also tried utf-8 and nothing at all
    headers: {
         "Accept": "application/octet-stream, text, text/plain, text/xml",
         "Accept-Encoding": "UTF-8",
         'Content-Type': "text/plain; charset=Cp1252;", // also tried utf-8, ISO-8859-1
         "User-Agent": "me"
    }
}).spread((res, body) => {
    body = body.toString();  // I also tried without toString();
    let ws = fs.createWriteStream('hhh.csv');
    ws.write(body);
    ws.end();

Whatever I do, umlauts are turned into .

Those are the headers the web service sends back:

'content-type': 'text; charset=Cp1252',
'content-length': '1895980',
vary: 'Accept-Encoding,User-Agent'

I'm trying this for days with no luck at all. What am I doing wrong?

Here's a list of question/answers that didn't solve my problem so far:

  • Umlauts broken when doing get request
  • Problems making http-post requests with German umlauts (ä ö ü) or special chars (ß) with Node.js
  • Send umlauts from Node.js to iOS
  • Automatic UTF-8 encoding in Node.js HTTP client
  • Custom HTTP header value - trying to pass umlaut characters
  • How to get UTF-8 in Node.js?
  • How to get i18next-node to display umlauts the right way?
  • How do I URl encode something in Node.js?

Can it be that one of the following causes my input string to not be UTF-8?

let hash = crypto.createHmac("sha256", this.options.Signer);
this.query['Signature'] = hash.update(stringToSign).digest("base64");

signer is a string containing 0-9, a-z, A-Z, +, and /.

this.query['Signature'] is part of the URL.

like image 548
baao Avatar asked Oct 05 '15 21:10

baao


1 Answers

I finally solved it, using iconv-lite and setting request's encoding¹ to null, making it return the body as a Buffer. Here is my now working configuration:

return request.getAsync({
        url,
        encoding: null,
        headers: {
            "Accept": "text, text/plain, text/xml",
            "Accept-Encoding": "UTF-8",
            'Content-Type': "text/plain; charset=utf-8;",
            "User-Agent": "me"
        }
    }).spread((res, body) => {
        let a = iconv.decode(new Buffer(body), 'cp1252');
        // now a is holding a string with correct Umlauts and ß
like image 182
baao Avatar answered Oct 13 '22 14:10

baao