How to read from a JSON file with unescaped backslashes?

Question

I have a JSON file that contains a connection string:

abc.json

{
  "host":"1.2.3.4",
  "user":"abc",
  "passwd":"s&]\yz$&u42/",
  "dbname":"sample",
  "port":2341
}

Here's my Python script which tries to read the data:

 import psycopg2 as pg
 dbconn = "C:\abc.json"
 with open(dbconn) as conn_file:
     conn = json.load(conn_file)

It gives me this error:

json.decoder.JSONDecodeError: Invalid \escape: line 4 column 16 (char 53)

How can I fix this error?

Answer 1

Your file is not valid: there's no such escape sequence as \y in JSON, and bare backslashes must themselves be escaped thus: \\.

The simplest solution, if you can, is to fix your file so that it is valid JSON, by escaping that backslash.

If for some reason you can't, it's possible to write a wrapper for json.loads() that catches this specific error and patches up the source text:

import json
from json.decoder import JSONDecodeError

def permissive_json_loads(text):
    while True:
        try:
            data = json.loads(text)
        except JSONDecodeError as exc:
            if exc.msg == 'Invalid \\escape':
                text = text[:exc.pos] + '\\' + text[exc.pos:]
            else:
                raise
        else:
            return data

For the sake of simplicity, the function above takes a string rather than a file.

It's also something of a "sledgehammer to crack a nut" approach, repeatedly trying to load the whole JSON document and fixing any unescaped backslashes as it finds them – which is reasonable for small JSON documents that rarely have the problem, but less appropriate if you're handling large JSON documents with lots of these unescaped backslash errors.

Here it is in action:

>>> print(text)
{
  "host":"1.2.3.4",
  "user":"abc",
  "passwd":"s&]\yz$&u42/",
  "dbname":"sample",
  "port":2341
}

>>> config = permissive_json_loads(text)
>>> print(config['passwd'])
s&]\yz$&u42/

In your case, you'd read from your file into a string, and call the function on that:

 dbconn = "C:\abc.json"
 with open(dbconn) as conn_file:
     conn_doc = conn_file.read()
 conn = permissive_json_loads(conn_doc)

Answer 2

Python >= 2.7.15 seems to produce the same error message as Python3:

test = """
{
  "host":"1.2.3.4",
  "user":"abc",
  "passwd":"s&]\yz$&u42/",
  "dbname":"sample",
  "port":2341
}
"""

print json.loads(test)

Error:

ValueError: Invalid \escape: line 5 column 16 (char 54)

Modifying @Zero Piraeus's code slightly makes it work under Python2.7:

import json
import re

def permissive_json_loads(text):
    _rePattern = re.compile(r'''(\d+)\)$''', re.MULTILINE)

    i = 0
    #  Make sure the loop is going to terminate.
    #  There wont be more iterations than the double amout of characters
    while True:
        i += 1
        if i > len(text) * 2:
            return
        try:
            data = json.loads(text)
        except ValueError, exc:
            exMsg = str(exc)
            if exMsg.startswith('Invalid \\escape'):
                m = re.search(_rePattern, exMsg)
                if not m:
                    return

                pos = int(m.groups()[0])
                print "Replacing at: %d" % pos
                text = text[:pos] + '\\' + text[pos:]
            else:
                raise
        else:
            return data

text = """
{
  "host":"1.2.3.4",
  "user":"abc",
  "passwd":"s&]\yz$&u42/",
  "dbname":"sample",
  "port":2341
}
"""

i = permissive_json_loads(text)
print i

Prints:

Replacing at position: 54
{u'passwd': u's&]\\yz$&u42/', u'host': u'1.2.3.4', u'port': 2341, u'user': u'abc', u'dbname': u'sample'}