zip() alternative for iterating through two iterables

Question

I have two large (~100 GB) text files that must be iterated through simultaneously.

Zip works well for smaller files but I found out that it's actually making a list of lines from my two files. This means that every line gets stored in memory. I don't need to do anything with the lines more than once.

handle1 = open('filea', 'r'); handle2 = open('fileb', 'r')

for i, j in zip(handle1, handle2):
    do something with i and j.
    write to an output file.
    no need to do anything with i and j after this.

Is there an alternative to zip() that acts as a generator that will allow me to iterate through these two files without using >200GB of ram?

Answer 1

You can use izip_longest like this to pad the shorter file with empty lines

in python 2.6

from itertools import izip_longest
with handle1 as open('filea', 'r'):
    with handle2 as open('fileb', 'r'): 
        for i, j in izip_longest(handle1, handle2, fillvalue=""):
            ...

or in Python 3+

from itertools import zip_longest
with handle1 as open('filea', 'r'), handle2 as open('fileb', 'r'): 
    for i, j in zip_longest(handle1, handle2, fillvalue=""):
        ...

Answer 2

itertools has a function izip that does that

from itertools import izip
for i, j in izip(handle1, handle2):
    ...

If the files are of different sizes you may use izip_longest, as izip will stop at the smaller file.