Error in looping through a text file in python

Question

I am trying to loop through a text file and apply some logic but I am not able to loop through the text file. So currently I have a text file that is structured like this:

--- section1 ---
"a","b","c"
"d","e","f"
--- section2 ---
"1","2","3"
"4","5","6"
--- section3 ---
"12","12","12"
"11","11","11"

I am trying to filter out the first line which contains '---' and convert the lines below into json until the next '---' line appear in the text document.

However I got this error " fields1 = next(file).split(',') StopIteration

with open(fileName,'r') as file:
    for line in file:
        if line.startswith('-') and 'section1' in line:
            while '---' not in next(file):
                fields1 = next(file).split(',')
                for x in range(0,len(fields1)):
                    testarr.append({
                    config.get('test','test'): fields1[x]           
                    })

                with open(test_dir,'w') as test_file:
                    json.dump(testarr, test_file)

Any idea why my code is not working or how i can solve the error ?

Answer 1

The cause of your error is that you are misusing the file object genrator by calling next on it twice as often as you think. Each call to next gets a line and returns it. Therefore, while '---' not in next(file): fields1 = next(file).split(',') gets a line, checks it for ---, then gets another line and tries to parse it. This means that you are able to skip a line containing the --- by having it come up in the second next. In that case you will get to the end of the file before you find the line you are looking for. StopIteration is how iterators normally indicate that their input has been exhausted.

There are a couple of other issues you may want to address in your code:

1. Using next on a generator like a file when you are already inside a for loop may cause undefined behavior. You may be getting away with it this time, but it is not good practice in general. The main reason you are getting away with it, by the way, is possibly that you never actually return control to the for loop once the while is triggered, and not that files are particularly permissive in this regard.
2. The inner with that dumps your data to a file is inside your while loop. That means that the file you open with 'w' permissions will get truncated for every iteration of the while (i.e., each line in the file). As the array grows, the output will actually appear fine, but you probably want to move that out of the inner loop.

The simplest solution would be to rewrite the code in two loops: one to find the start of the part you care about, and the other to process it until the end is found.

Something like this:

test_arr = []
with open(fileName, 'r') as file:
    for line in file:
        if line.startswith('---') and 'section1' in line:
            break

    for line in file:
        if '---' in line:
            break
        fields1 = line.split(',')
        for item in fields1:
            testarr.append({config.get('test','test'): item})

with open(test_dir,'w') as test_file:
    json.dump(testarr, test_file)

EDIT:

Taking @tripleee's advice, I have removed the regex check for the start line. While regex gives great precision and flexibility for finding a specific pattern, it is really overkill for this example. I would like to point out that if you are looking for a section other than section1, or if section1 appears after some other lines with dashes, you will absolutely need this two-loop approach. The one-loop solutions in the other answers will not work in a non-trivial case.