Ideal Chunk Size for python requests

Question

Is there any guideline on selecting chunk size?

I tried different chunk size but none of them give download speed comparable to browser or wget download speed

here is snapshot of my code

 r = requests.get(url, headers = headers,stream=True)  total_length = int(r.headers.get('content-length'))  if not total_length is None: # no content length header  for chunk in r.iter_content(1024):      f.write(chunk) 

Any help would be appreciated.?

Edit: I tried network with different speed.. And I am able to achieve higher speed than my home network.. But when I tested wget and browser.. Speed is still not comparable

Thanks

Answer 1

You will lose time switching between reads and writes, and the limit of the chunk size is AFAIK only the limit of what you can store in memory. So as long as you aren't very concerned about keeping memory usage down, go ahead and specify a large chunk size, such as 1 MB (e.g. 1024 * 1024) or even 10 MB. Chunk sizes in the 1024 byte range (or even smaller, as it sounds like you've tested much smaller sizes) will slow the process down substantially.

For a very heavy-duty situation where you want to get as much performance as possible out of your code, you could look at the io module for buffering etc. But I think increasing the chunk size by a factor of 1000 or 10000 or so will probably get you most of the way there.