OverflowError: Unsupported UTF-8 sequence length when > encoding string

Question

Inside a Twisted Resource, I am returning a json encoded dict as the response var below. The data is a list of 5 people with name, guid, and a couple other fields less than 32 characters long each, so not a ton of data.

I get this OverflowError exception pretty often, but I don't quite understand what the unsupported utf-8 sequence length refers to.

self.request.write(ujson.dumps(response))

exceptions.OverflowError: Unsupported UTF-8 sequence length when encoding string

Answer 1

Just a note that I recently encountered this same error, and can give a little background.

If you see this, it's possible you're trying to json encode a Mongo Object with ujson in python.

Using the native python library, we get a more helpful error message:

TypeError: ObjectId('510652d322fc956ca9e41342') is not JSON serializable

ujson is somehow trying to parse an ObjectId python object and getting lost. There are a few options, the most direct being wiping the '_id' field from Mongo before saving. You could also subclass ujson to somehow parse or munge the ObjectIds into a simple character string.