"Out of Memory" error with mechanize

Question

I was trying to scrape some information from a website page by page, basically here's what I did:

import mechanize
MechBrowser = mechanize.Browser()

Counter = 0

while Counter < 5000:
    Response = MechBrowser.open("http://example.com/page" + str(Counter))
    Html = Response.read()
    Response.close()

    OutputFile = open("Output.txt", "a")
    OutputFile.write(Html)
    OutputFile.close()

    Counter = Counter + 1

Well, the above codes ended up throwing out "Out of Memory" error and in task manager it shows that the script used up almost 1GB memory after several hours running... how come?!

Would anybody tell me what went wrong?

Answer 1

This is not exactly a memory leak, but rather an undocumented feature. Basically, mechanize.Browser() is collectively storing all browser history in memory as it goes.

If you add a call to MechBrowser.clear_history() after Response.close(), it should resolve the problem.