Serializing output to JSON - ValueError: Circular reference detected

Question

I'm trying to output results of my mysql query to JSON. I have problem with serializing datetime.datetime field, so I wrote small function to do that:

def date_handler(obj):     if hasattr(obj, 'isoformat'):         return obj.isoformat()     else:         return obj 

and then in main code I'm just running:

products_json = [] for code in best_matching_codes:     cur = db.cursor()     query = "SELECT * FROM %s WHERE code LIKE '%s'" % (PRODUCTS_TABLE_NAME, product_code)     cur.execute(query)     columns = [desc[0] for desc in cur.description]     rows = cur.fetchall()     for row in rows:         products_json.append(dict((k,v) for k,v in zip(columns,row)))     return json.dumps(products_json, default = date_handler) 

However, since I wrote date_handler function, I'm getting "ValueError: Circular reference detected"

127.0.0.1 - - [10/Jan/2013 00:42:18] "GET /1/product?code=9571%2F702 HTTP/1.1" 500 - Traceback (most recent call last):   File "/Library/Python/2.7/site-packages/flask/app.py", line 1701, in __call__ return self.wsgi_app(environ, start_response)   File "/Library/Python/2.7/site-packages/flask/app.py", line 1689, in wsgi_app response = self.make_response(self.handle_exception(e))   File "/Library/Python/2.7/site-packages/flask/app.py", line 1687, in wsgi_app response = self.full_dispatch_request()   File "/Library/Python/2.7/site-packages/flask/app.py", line 1360, in full_dispatch_request rv = self.handle_user_exception(e)   File "/Library/Python/2.7/site-packages/flask/app.py", line 1358, in full_dispatch_request rv = self.dispatch_request()   File "/Library/Python/2.7/site-packages/flask/app.py", line 1344, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args)   File "/Users/pisarzp/Desktop/SusyChoosy/susyAPI/test1.py", line 69, in product_search return json.dumps(products_json, default = date_handler)   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 238, in dumps **kw).encode(obj)   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 201, in encode chunks = self.iterencode(o, _one_shot=True)   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 264, in iterencode return _iterencode(o, 0) ValueError: Circular reference detected 

What did I break? Is there a better way to serialize output to JSON?

Answer 1

The function you pass as the default argument will only be called for objects that are not natively serializable by the json module. It must return a serializable object, or raise a TypeError.

Your version returns the same object you were passed if it's not of the one type you're fixing (dates). That is causing the circular reference error (which is misleading, since the circle is between one object and itself after being processed by date_handler).

You can start to fix this by changing date_handler to raise an exception in its else block. That will still probably fail, but you can probably then find out what object it is that is in your data structure causing the problem using code like this:

def date_handler(obj):     if hasattr(obj, 'isoformat'):         return obj.isoformat()     else:         raise TypeError(             "Unserializable object {} of type {}".format(obj, type(obj))         ) 

Answer 2

Instead of raising the TypeError yourself, you should relay the call to JSONEncoder's default-method:

def date_handler(obj):     if hasattr(obj, 'isoformat'):         return obj.isoformat()     else:         json.JSONEncoder.default(self,obj) 

This will also raise TypeError and is a better practice, it allows for JSONEncoder to try and encode the type your method can't.