Python (3.5) - urllib.request.urlopen - Progress Bar Available?

Question

I'm trying to search the world wide web for this answer, but I feel there answer may be no. I'm using Python 3.5 and a library called urllib.request with a method called urllib.request.urlopen(url) to open a link and download a file.

It would be nice to have some kind of measure of progress for this, as the file(s) are over 200MB. I'm looking at the API here, and don't see any kind of parameter with a hook.

Here's my code:

downloadURL = results[3] #got this from code earlier up
rel_path = account + '/' + eventID + '_' + title + '.mp4'

filename_abs_path = os.path.join(script_dir, rel_path)
print('>>> Downloading >>> ' + title)

# Download .mp4 from a url and save it locally under `file_name`:
with urllib.request.urlopen(downloadURL) as response, open(filename_abs_path, 'wb') as out_file:
    shutil.copyfileobj(response, out_file)

Can anyone provide insight if they think I can potentially have a progress bar or would the only way be to use a different library? I'm looking to keep the code quite short and simple, I just need some indication of the file being downloaded. Thanks for any help provided!

Answer 1

If the response includes a content-length you can read the incoming data in blocks and calculate percent done. Unfortunately, web servers that "chunk" responses don't always provide a content length, so it doesn't always work. Here is an example to test.

import urllib.request
import sys
import io

try:
    url = sys.argv[1]
except IndexError:
    print("usage: test.py url")
    exit(2)

resp = urllib.request.urlopen(url)
length = resp.getheader('content-length')
if length:
    length = int(length)
    blocksize = max(4096, length//100)
else:
    blocksize = 1000000 # just made something up

print(length, blocksize)

buf = io.BytesIO()
size = 0
while True:
    buf1 = resp.read(blocksize)
    if not buf1:
        break
    buf.write(buf1)
    size += len(buf1)
    if length:
        print('{:.2f}\r done'.format(size/length), end='')
print()

Answer 2

@tdelaney's answer is great, but in Python 3.8 you have to use getvalue() method instead of read():

    import io, urllib.request

    with urllib.request.urlopen(Url) as Response:
        Length = Response.getheader('content-length')
        BlockSize = 1000000  # default value

        if Length:
            Length = int(Length)
            BlockSize = max(4096, Length // 20)

        print("UrlLib len, blocksize: ", Length, BlockSize)

        BufferAll = io.BytesIO()
        Size = 0
        while True:
            BufferNow = Response.read(BlockSize)
            if not BufferNow:
                break
            BufferAll.write(BufferNow)
            Size += len(BufferNow)
            if Length:
                Percent = int((Size / Length)*100)
                print(f"download: {Percent}% {Url}")

        print("Buffer All len:", len(BufferAll.getvalue()))