Python3 json.dumps gives TypeError: keys must be a string

Question

I've got a simple web server written in Python3 (using classes from http.server) that I'm porting from 2 to 3.

I have the following code:

# More code here...
postvars = cgi.parse_qs(self.rfile.read(length),
                        keep_blank_values=1)
something.json_run(json.dumps(postvars))

Which throws:

TypeError: keys must be a string

By inspecting the data, I've determined that parse_qs seems to encode the keys as bytes, which is what's throwing the error (json doesn't like bytes, apparently).

import json
json.dumps({b'Throws error' : [b"Keys must be a string"]})
json.dumps({'Also throws error': [b'TypeError, is not JSON serializable']})
json.dumps({'This works': ['No bytes!']})

What is the best solution here? With Python 2, the code works fine because parse_qs uses str instead of bytes. My initial thought is that I probably need to write a JSON serializer. Not that it's difficult for something so simple, but I'd prefer not to if I can do it some other way (e.g. translate the dictionary to using strings instead of bytes).

Answer 1

Firstly, the cgi.parse_qs function is deprecated and merely an alias for urllib.parse.parse_qs, you may want to adjust your import path.

Secondly, you are passing in a byte string into the parse method. If you pass in a regular (unicode) string instead the parse_qs method returns regular strings:

>>> from urllib.parse import parse_qs
>>> parse_qs(b'a_byte_string=foobar')
{b'a_byte_string': [b'foobar']}
>>> parse_qs('a_unicode_string=foobar')
{'a_unicode_string': ['foobar']}

So you'll need to decode your file-read byte string to a regular string first.