PayPal Payflow Gateway UTF-8 Characters

Question

I'm having a problem getting the PayPal Payflow Gateway accept my HTTPS POST with non US-ASCII characters in it. Not matter what I post with these special characters, it seems to want to accept only US-ASCII encoded bytes. If I send UTF-8 encoded bytes, it still works, but fails to parse some of the request NVP values. I know it is possible because I've posted with another developer's test page to my account (http://ccaples.com/ NVP Quick Test) and that post seems to preserve the special characters.

Here's an example. I've tried with MANY different Content-Type headers and Accept/Accept-Charset values including specifying "; charset=UTF-8" or "; charset=utf-8" in the Content-Type after the text/namevalue.

POST https://pilot-payflowpro.paypal.com/ HTTP/1.1
Content-Type: text/namevalue
User-Agent: KLMS Payflow API for Java
X-VPS-Request-ID: 36A4ED051A8B492ABF70E6BE51CB13D5
X-VPS-CLIENT-TIMEOUT: 20
Connection: close
X-VPS-VIT-INTEGRATION-PRODUCT: KLMS Payflow API for Java
X-VPS-VIT-INTEGRATION-VERSION: 2.0.008
X-VPS-VIT-PROXY: Y
X-VPS-VIT-RUNTIME-VERSION: 20.45-b01
X-VPS-VIT-OS-ARCHITECTURE: amd64
X-VPS-VIT-OS-VERSION: 6.1
X-VPS-VIT-OS-NAME: Windows 7
Cache-Control: no-cache
Pragma: no-cache
Host: pilot-payflowpro.paypal.com
Content-Length: 694

USER[8]=XXXXXXXX&VENDOR[13]=XXXXXXXXXXXXX&PARTNER[6]=PayPal&PWD[8]=XXXXXXXX&VERBOSITY[4]=HIGH&BILLTOEMAIL[28][email protected]&TRXTYPE[1]=A&TENDER[1]=C&ACCT[16]=4111111111111111&EXPDATE[4]=1117&CVV2[3]=123&BILLTOFIRSTNAME[4]=Josè&BILLTOLASTNAME[7]=Elkjærd&BILLTOSTREET[14]=123 Elm Street&BILLTOCITY[9]=Elm Creek&BILLTOSTATE[2]=VA&BILLTOZIP[5]=22203&BILLTOCOUNTRY[2]=US&BILLTOPHONENUM[12]=763-221-5593&CUSTBROWSER[108]=Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36&CUSTHOSTNAME[12]=192.168.1.10&CUSTIP[12]=192.168.1.10&AMT[6]=600.00&CURRENCY[3]=USD&COMMENT1[26]=Law Enforcement Curriculum&COMMENT2[16]=Standard Version

HTTP/1.1 200 OK
Connection: close
Server: VPS-3.033.00
X-VPS-Request-ID: 36A4ED051A8B492ABF70E6BE51CB13D5
Date: Sun, 25 Jan 2015 17:45:05 GMT
Content-type: text/namevalue
Content-length: 199

RESULT=104&PNREF=B70P7B6EBAE7&RESPMSG=Timeout waiting for Processor response&TRANSTIME=2015-01-25 09:44:47&BILLTOFIRSTNAME=Jos &BILLTOLASTNAME=NotProvided&AMT=600.00&ACCT=1111&EXPDATE=1117&CARDTYPE=0

Ignore the 104 timeout for now. PayPal is working on that. What you can see is that the BILLTOFIRSTNAME is Jos(weird character that's not è) and the æ character in BILLTOLASTNAME must have caused the server's NVP parsing algorithm to have completely failed to parse because it says NotProvided.

Here's the code that converts the Java string in NVP formatted form to bytes:

        final byte[] requestBytes = requestString.getBytes(Charsets.UTF_8);

Any idea what character encoding the server-side NVP parser is looking for by default, or how to tell it what character encoding I'm sending the body of the POST in?

Answer 1

I have found the solution. It appears that it is not supported by the Payflow Gateway Docs. The docs seem to be out of date and need updating. I worked with PayPal support and it does appear that the Payflow Gateway supports UTF-8. Here is the trick: either you need to NOT specify the length parameter in the name/value pairs you submit (i.e. BILLTOFIRSTNAME=Josè) or you have to specify in the length parameter as not the length in CHARACTERS (i.e. not value.length()), but the actual length in BYTES of the UTF-8 encoded byte stream for the value (i.e. value.getBytes("UTF-8").length). This transaction worked, stored the UTF-8 characters and displayed properly in the PayPal Manager, and the proper value was returned in the name/value pairs of the Payflow Gateway response. As you can see, the difference from above is that the BILLTOFIRSTNAME and BILLTOLASTNAME length tags are now larger and specify the length in bytes instead of the length in characters. Josè [4 characters but 5 bytes: 4A 6F 73 C3 A8] and Elkjærd & Grün [14 characters but 16 bytes: 45 6C 6B 6A C3 A6 72 64 20 26 20 47 72 C3 BC 6E].

POST https://pilot-payflowpro.paypal.com/ HTTP/1.1
Content-Type: text/namevalue
Connection: close
User-Agent: KLMS Payflow for Java/2.0.008
X-VPS-Request-ID: B8DABD9BDFE246EC909B4CF741030133
X-VPS-CLIENT-TIMEOUT: 20
X-VPS-VIT-INTEGRATION-PRODUCT: KLMS Payflow for Java
X-VPS-VIT-INTEGRATION-VERSION: 2.0.008
X-VPS-VIT-PROXY: Y
X-VPS-VIT-RUNTIME-VERSION: 20.45-b01
X-VPS-VIT-OS-ARCHITECTURE: amd64
X-VPS-VIT-OS-VERSION: 6.1
X-VPS-VIT-OS-NAME: Windows 7
Cache-Control: no-cache
Pragma: no-cache
Host: pilot-payflowpro.paypal.com
Content-Length: 771

USER[8]=XXXXXXXX&VENDOR[13]=XXXXXXXXXXXXX&PARTNER[6]=PayPal&PWD[8]=XXXXXXXX&VERBOSITY[4]=HIGH&BILLTOEMAIL[28][email protected]&TRXTYPE[1]=A&TENDER[1]=C&ACCT[16]=4111111111111111&EXPDATE[4]=0216&CVV2[3]=123&BILLTOFIRSTNAME[5]=Josè&BILLTOLASTNAME[16]=Elkjærd & Grün&BILLTOSTREET[14]=123 Elm Street&BILLTOSTREET2[10]=Elm & Vine&BILLTOCITY[9]=Elm Creek&BILLTOSTATE[2]=VA&BILLTOZIP[5]=12345&BILLTOCOUNTRY[2]=US&BILLTOPHONENUM[12]=763-221-5593&CUSTBROWSER[108]=Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36&CUSTHOSTNAME[12]=192.168.1.10&CUSTIP[12]=192.168.1.10&AMT[6]=600.00&CURRENCY[3]=USD&COMMENT1[45]=Victim Advocate Curriculum (Standard Version)&COMMENT2[36]=Purchased for J.C. Hamlin (jchamlin)

HTTP/1.1 200 OK
Connection: close
Server: VPS-3.033.00
X-VPS-Request-ID: B8DABD9BDFE246EC909B4CF741030133
Date: Wed, 28 Jan 2015 01:45:29 GMT
Content-type: text/namevalue
Content-length: 209

RESULT=104&PNREF=B10P7D2F1643&RESPMSG=Timeout waiting for Processor response&TRANSTIME=2015-01-27 17:45:12&BILLTOFIRSTNAME=Josè&BILLTOLASTNAME[16]=Elkjærd & Grün&AMT=600.00&ACCT=1111&EXPDATE=0216&CARDTYPE=0