Python truncating international string

Question

I've been trying to debug this for far too long, and I obviously have no idea what I'm doing, so hopefully someone can help. I'm not even sure what I should be asking, but here it goes:

I'm trying to send Apple Push Notifications, and they have a payload size limit of 256 bytes. So subtract some overhead stuff, and I'm left with about 100 english characters of main message content.

So if a message is longer than the max, I truncate it:

MAX_PUSH_LENGTH = 100
body = (body[:MAX_PUSH_LENGTH]) if len(body) > MAX_PUSH_LENGTH else body

So that's fine and dandy, and no matter how long of a message I have (in english), the push notification sends successfully. However, now I have an Arabic string:

str = "هيك بنكون 
عيش بجنون تون تون تون هيك بنكون 
عيش بجنون تون تون تون 
أوكي أ"

>>> print len(str)
109

So that should truncate. But, I always get an invalid payload size error! Curious, I kept lowering the MAX_PUSH_LENGTH threshold to see what it would take for it to succeed, and it's not until I set the limit to around 60 that the push notification succeeded.

I'm not exactly sure if this has something to do with the byte size of languages other than english. It is my understanding that an English character takes one byte, so does an Arabic character take 2 bytes? Might this have something to do with it?

Also, the string is JSON encoded before it is sent off, so it ends up looking something like this: \u0647\u064a\u0643 \u0628\u0646\u0643\u0648\u0646 \n\u0639\u064a\u0634 ... Could it be that it is being interpreted as a raw string, and just u0647 is 5 bytes?

What should I be doing here? Are there any obvious errors or am I not asking the right question?

Answer 1

If you have a python unicode value and you want to truncate, the following is a very short, general, and efficient way to do it in Python.

def truncate_unicode_to_byte_limit(src, byte_limit, encoding='utf-8'):
    '''
    truncate a unicode value to fit within byte_limit when encoded in encoding

    src: a unicode
    byte_limit: a non-negative integer
    encoding: a text encoding

    returns a unicode prefix of src guaranteed to fit within byte_limit when
    encoded as encoding.
    '''
    return src.encode(encoding)[:byte_limit].decode(encoding, 'ignore')

So for example:

s = u"""
    هيك بنكون
    ascii
    عيش بجنون تون تون تون هيك بنكون
    عيش بجنون تون تون تون
    أوكي أ
"""

b = truncate_unicode_to_byte_limit(s, 73)
print len(b.encode('utf-8')), b

produces output:

73 
    هيك بنكون
    ascii
    عيش بجنون تون تون تو