"Memory Leak" with grequests?

Question

This is a stripped down version of the script that causes continually increasing memory usage, I've seen it go past 600MB after 2 minutes:

import requests
import grequests

lines = (grequests.get(l.strip(), timeout=15) for l in open('links.txt') if len(l.strip()))

for r in grequests.imap(lines, size=20):
    if r.ok:
        print r.url

links.txt is a file containing a large number of urls, the problem happens with several large groups of urls that I have collected. It seems to me like that response objects may not be being deferenced?

I updated gevent, requests and grequests today, here are their versions:

In [2]: gevent.version_info
Out[2]: (1, 0, 0, 'beta', 3)

In [5]: requests.__version__
Out[5]: '0.13.5'

grequests doesn't have a version number that I could find.

Thanks in advance for any answers.

Answer 1

This answer is just an alias and link back for people who might need this link.

I use the imap function and requests.Session to reduce the memory usage while making 380k requests in my scripts.