Special characters in REST request

Question

I am developing an API using CodeIgniter, and Phils RESTserver. I am trying to send a POST request containing special characters, but the string is not added to the database.

CodeIgniter also says that lastname is required (that it is not present in the string). Why?

I am using this format:

application/x-www-form-urlencoded

This is my string:

firstname=Andrew&lastname=Åsberger

It is very important that I can use special characters for internationalization.

Thankful for all input!

Answer 1

You should URI-encode each name and value. Hopefully the client and server code will both agree that UTF-8 should be used for encoding the octets of characters outside of the US-ASCII range (since earlier URI-encoding standards weren't specific and there is legacy code out there that tries other encodings), so your example becomes:

firstname=Andrew&lastname=%C3%85sberger

Just like it would in the query portion of a URI used with a GET.

Answer 2

It seems like you are having an encoding issue. You need to make sure that you are using UTF8 from end to end: client (browser), server (PHP), db connection and db. I assume your db table(s) are already UTF8, but what many forget is the connection to the database. Right after you connect to the database, you should run the "query" SET NAMES UTF8. Not sure if CodeIgniter uses the db connection to escape characters.

I don't use CodeIgniter, but if it's not using the proper encoding, then double-byte characters get expanded out into 2 characters. For example, if you running urlencode('Å') returns %C3%85, not %C5. This is actually a SQL injection method. If one of the characters it "decodes" to is a ' or ", then there is a quoting issue/vulnerability. This could cause CodeIgniter to evaluate the string incorrectly.

Finally, are you doing your POST through javascript? Javascript does not support UTF8 encoding, so it causes some problems depending on how you POST. You can use javascript to POST a html form, but you can run into problems when you try to do an ajax post using strings you make yourself. Although unescape( encodeURIComponent( s ) ) supposedly works.

Answer 3

Once i had a similar issue while inserting products with special chars in name into cart and in creating my urls

Not sure, but it may be helpful from another point of view. I also had added a my_url_helper in addition for my project to handle urls. mb_string handles char replacements very well. Sorry for my bad language. :(
File: application/config.php

/*
|--------------------------------------------------------------------------
| Allowed URL Characters
|--------------------------------------------------------------------------
|
| This lets you specify with a regular expression which characters are permitted
| within your URLs.  When someone tries to submit a URL with disallowed
| characters they will get a warning message.
|
| As a security measure you are STRONGLY encouraged to restrict URLs to
| as few characters as possible.  By default only these are allowed: a-z 0-9~%.:_-
|
| Leave blank to allow all characters -- but only if you are insane.
|
| DO NOT CHANGE THIS UNLESS YOU FULLY UNDERSTAND THE REPERCUSSIONS!!
|
*/  

//This is not default, its modified for turkish chars
$config['permitted_uri_chars'] = 'a-üöçşığz A-ÜÖÇŞİĞZ 0-9~%.:_\-';