ValueErrors, ProtocolErrors and ChunkedEncodingErrors when using requests.get().json()

Question

I am working in Python on gathering data (trading volume, mining fees etc.) from the Bitcoin blockchain. To do this, I am trying to JSON query various pages on blockchain.info.

After importing json, requests and urlopen, I start my code with:

url = "https://blockchain.info/rawblock/00000000000000000000427c2cbfd8868c5bc987603d2483c9637f052316f89f?"
currentblock = requests.get(url, stream = True).json()
prevhash = str(currentblock['prev_block'])

I choose to start my code on the most recent block (everything after /rawblock/ in the url is the id of the block) and then from the output of requests.get(url, stream=True).json(), I get the id of the previous block in the chain (to then query in the next loop/cycle).

I go into a for-loop, where I repeat the process of gathering info from each block, getting the id of the previous block and then continuing to query backwards in that way:

prevhash = str(currentblock['prev_block'])
for num in range(0,2700):
    currenthash = prevhash
    url = "https://blockchain.info/rawblock/" + currenthash + "?"
    print(currenthash)
    try:
        currentblock = requests.get(url, stream = True).json()
    except ValueError:
        print(hashnum)
        print("Response content is not JSON")
    prevhash = currentblock['prev_block']

The reason I set up the try,except sequence is because at seemingly random intervals, I would get an error that would stop my program which said something the response from requests.get() isn't JSON. I found that if I simply catch that error, the program queries the webpage again and most times, on the second try, everything works out fine and the program continues running.

However, I just encountered what looks like a nasty string of errors and to be honest, I have no idea what they mean or how to fix them.

Here is the traceback (with the intermediate steps taken out):

Traceback (most recent call last):
  File "C:\Users\Vlad\Desktop\lib\site-packages\urllib3\response.py", line 685, in _update_chunk_length
    self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  ...
  File "C:\Users\Vlad\Desktop\lib\site-packages\urllib3\response.py", line 689, in _update_chunk_length
    raise httplib.IncompleteRead(line)
http.client.IncompleteRead: IncompleteRead(0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  ...
  File "C:\Users\Vlad\Desktop\lib\site-packages\urllib3\response.py", line 443, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  ...
  File "C:\Users\Vlad\Desktop\lib\site-packages\requests\models.py", line 753, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

The thing that confuses me most is that these errors are not consistent (that's to say, if I try the exact same code, I won't get errors in the same times/places as before). This is my first time working with JSON queries so I'm not familiar with it at all, but I don't know why the errors happen at seemingly random intervals.

Out of curiosity, I restarted my program at the point in the chain right before the above error occurred - and there was no error this time. It just kept running without a problem.

Any help would be enormously appreciated and please let me know if I can explain more in detail what I'm doing!

Thank you

Answer 1

Use the retry package to retry requests that fail due to network errors.

import requests
from retry import retry


@retry(exceptions=Exception, tries=5, delay=1)
def get_currentblock(url):
    return requests.get(url)


url = "https://blockchain.info/rawblock/00000000000000000000427c2cbfd8868c5bc987603d2483c9637f052316f89f"
currentblock = get_currentblock(url).json()
prevhash = str(currentblock['prev_block'])
for num in range(0,2700):
    currenthash = prevhash
    url = "https://blockchain.info/rawblock/" + currenthash
    print(currenthash)
    try:
        currentblock = get_currentblock(url).json()
    except ValueError:
        print(hashnum)
        print("Response content is not JSON")
    prevhash = currentblock['prev_block']

This will retry the get() up to five times at one second delays upon any Exception. You may want to change exceptions=Exception to be more restrictive as you get a better idea of the exceptions you may encounter. The get_currentblock() method just returns the response rather than the JSON component in order to avoid triggering a retry on a JSON parsing error, i.e. we only wish to retry when a network error occurs.

Also notice that stream=True has been removed from the get() call because it has no effect in conjunction with the json() call since the entire response must be received before attempting to parse the content as a JSON object. Streaming requests are used for getting an iterator over the response content in order to process each line as it's received, but that's not how it's being used here.

Lastly, I removed the ? from the end of the URL. The ?signifies following query parameters, but is unnecessary since there are no query parameters.