Python HTTP Request Exception: LineTooLong

Question

In pyhton, When I am trying to get json data from an URL I got the following exception.

('Connection aborted.', LineTooLong('got more than 65536 bytes when reading header line'))

But when I try curl it works without problem.

curl "http://ccmixter.org/api/query?f=json&datasource=default&lic=pd&sort=id&limit=20&offset=20"

Here is the sample python code which raise the exception:

import requests

url = "http://ccmixter.org/api/query?f=json&datasource=default&lic=pd&sort=id&limit=20&offset=20"
req = requests.get(url)
print(req.content)

Answer 1

This is caused by the http.client module, which has some limits on the header size.

You have two options :

• You can manually change _MAXLINE to a higher value, knowing that the header size will not exceed it.

    import http.client
  
    http.client._MAXLINE = 655360
  
    url = "http://ccmixter.org/api/query?f=json&datasource=default&lic=pd&sort=id&limit=20&offset=20"
    req = requests.get(url)
    print(req.content)
  

• If you don't want to worry about the header size, you can override the parse_headers function from the http.client module with a custom one in which I removed the part of the code that raises the exception once the limit is exceeded.

    def disable_header_limit():
        import http.client
        import email.parser
        from http.client import HTTPMessage, _MAXLINE
  
        def parse_headers(fp, _class=HTTPMessage):
            headers = []
            while True:
                line = fp.readline(_MAXLINE + 1)
                headers.append(line)
                if line in (b'\r\n', b'\n', b''):
                    break
            hstring = b''.join(headers).decode('iso-8859-1')
            return email.parser.Parser(_class=_class).parsestr(hstring)
  
        http.client.parse_headers = parse_headers
  
    disable_header_limit();
    url = "http://ccmixter.org/api/query?f=json&datasource=default&lic=pd&sort=id&limit=20&offset=20"
    req = requests.get(url)
    print(req.content)