Issue with character encoding in POST requests sent with Firefox

Question

Recently I've come across some very strange behavior related to character encoding for AJAX calls made using the POST method. To make a long story short, I have an HTML form with text fields that can accept diacritics (e.g. "ä"). When the form is submitted, the form data is wrapped in an XML block and sent to a server, which stores that information in a MySQL database. Subsequently, that information is retrieved from the database and displayed to regular users, as is.

If the request is sent from Chrome or IE, everything is fine. This means that the data, including the diacritics, is sent, stored, then retrieved and displayed correctly. However, when I use Firefox for this, the XML appears to submit the form data right, but when I reload the web page, the previously sent diacritics don't appear. In other words, they seem to get lost somewhere along the way. For example, if the XML contains the word "tästä", when I load the page I see "tst".

Why is this happening? Is Firefox encoding the post messages differently from IE and Chrome?

In case it helps, I've attached the request and response headers from Chrome and Firefox, for exactly the same form content - only one example:

By the way, I'm not encoding the data before sending it to the server, just simply retrieving the value of the form fields, as is.

CHROME:

The XML data block:

<request>
<session>{hidden by me}</session>
<builder>Hem i Stan tästä</builder>
</request>

The request headers:

Accept:*/*
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-US,en;q=0.8
Connection:keep-alive
Content-Length:562
Content-Type:application/x-www-form-urlencoded
Cookie:PHPSESSID=rlne2d787j0np52ec5rtn04dm1
Host:83.150.87.220
Origin:http://hidden.by.me
Referer:http://http://hidden.by.me/?c=2094211
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1
X-Requested-With:XMLHttpRequest

The response headers:

Connection:Keep-Alive
Content-Encoding:gzip
Content-Type:application/xml
Date:Mon, 17 Sep 2012 16:21:58 GMT
Keep-Alive:timeout=5, max=100
Server:Apache/2.2.11 (Win32) PHP/5.2.9-1
Transfer-Encoding:chunked
Vary:Accept-Encoding

FIREFOX:

The XML data block:

<request>
<session>{hidden by me}</session>
<builder>Hem i Stan tästä</builder>
</request>

The request headers:

Accept  */*
Accept-Encoding gzip, deflate
Accept-Language en-us,en;q=0.5
Connection  keep-alive
Content-Length  562
Content-Type    application/x-www-form-urlencoded; charset=UTF-8
Cookie  PHPSESSID=kvfg4fp2trorllim19dmn241c7
Host    hidden.by.me
Referer http://hidden.by.me/?c=2094211
User-Agent  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:14.0) Gecko/20100101 Firefox/14.0.1
X-Requested-With    XMLHttpRequest

The response headers:

Connection  Keep-Alive
Content-Encoding    gzip
Content-Type    application/xml
Date    Mon, 17 Sep 2012 16:21:23 GMT
Keep-Alive  timeout=5, max=100
Server  Apache/2.2.11 (Win32) PHP/5.2.9-1
Transfer-Encoding   chunked
Vary    Accept-Encoding

Answer 1

As @Pointy mentioned some time ago, the problem was related to the Content-Type of the POST request, because Firefox appears to encode POST messages differently than other browsers. In my head, Data-Type and Content-Type were the same and so, I didn't realize it's necessary to specify UTF-8 as the character encoding standard in both cases. But once I had changed both the Content-Type and the Data-Type as well to a clear "text/xml; charset=UTF-8", the problem was resolved.

Answer 2

I am soooo happy. Thank you guys for posting and figuring this out earlier. It took me a couple hours to get close enough to the problem to find this through googling, but because of your comments, I got this solved in less than a day; and in time for the big presentation tomorrow! :)

It was so bizarre, seeing that all browsers were sending the very same data string in an AJAX request but getting different results, depending on the browser (Firefox being different.)

I tried this, but it didn't work:

req.setRequestHeader ("encoding", "utf-8");

Then I just did what you said Firefox does and one coding solution works in all browsers.

req.setRequestHeader("Content-type", "application/x-www-form-urlencoded;charset=utf-8");

I've tested on Chrome, MSIE, Firefox, Safari, Opera and Opera Next. Works every time!