Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Custom generator for file filter

Tags:

python

I'm writing a small wrapper class around open that will filter out particular lines from a text file and then split them into name/value pairs before passing them back to the user. Naturally, this process lends itself to being implemented using generators.

My "file" class

class special_file:
    def __init__(self, fname):
        self.fname = fname

    def __iter__(self):
        return self

    def __next__(self):
        return self.next()

    def next(self):
        with open(self.fname, 'r') as file:
            for line in file:
                line = line.strip()
                if line == '':
                    continue
                name,value = line.split()[0:2]
                if '%' in name:
                    continue
                yield name,value
            raise StopIteration()

Userland code

for g in special_file('input.txt'):
    for n,v in g:
        print(n,v)

My code, sadly, has two enormous problems: 1) special_file returns a generator when it really needs to return a tuple, and 2) the StopIteration() exception is never raised so the file is read repeatedly ad infinitum. I have a sneaking suspicion that these two issues are related, but my understanding of generators and iterable sequences is fairly limited. Have I missed something painfully obvious about implementing a generator?

Edit:

I fixed my infinite reading problem by moving the first generator outside of the loop and then just looping over it.

g = special_file('input.txt')
k = next(g)
for n,v in k:
    print(n,v)

However, I would like the user to be able to use it like a normal call to open:

for n,v in special_file('input.txt'):
    print(n,v)
like image 635
Tim Avatar asked May 15 '16 07:05

Tim


1 Answers

You've implemented an iterator, in terms of using a generator. Just write the generator directly.

def special_file(filename):
    with open(filename, 'r') as file:
        for line in file:
            line = line.strip()
            if line == '':
                continue
            name, value, *_ = line.split()
            if '%' in name:
                continue
            yield name, value

See here for an overview of what it means to be iterable, what an iterator is, and python's protocols for using them.

like image 176
Jeff Mercado Avatar answered Oct 06 '22 00:10

Jeff Mercado