Can anyone explain the root of this index out of range error?

Question

The following code throws up a mysterious error that I cannot find the solution to. It works fine when I tested it in a bigger module, so cannot see why this doesn't work:

Code

import csv

with open('studentinfo.txt','a') as fo: #open the file in append mode (add to file, we don't wish to overwrite!)
        studentfileWriter=csv.writer(fo) #fo = file out (this can be called anything you like)
        id=input("Enter Student Id:")
        firstname=input("Enter firstname:")
        surname=input("Enter Surname:")
        test1=input("Enter test1 score:")
        test2=input("Enter test2 score:")
        test3=input("Enter test3 score:")
        studentfileWriter.writerow([id,firstname,surname,"Test1:"+test1,"Test2:"+test2,"Test3:"+test3])
        print("Record has been written to file")


with open("studentinfo.txt", "r") as f:
    reader = csv.reader(f)
    sorted_list = list(reader)  # turn the reader iterator into a list
    sorted_list.sort(key=lambda x: x[2])  # use the third column as a sorting key
    print("\n".join(str(row) for row in sorted_list))  # prettier print

Error Message

sorted_list.sort(key=lambda x: x[2])  # use the third column as a sorting key
IndexError: list index out of range

It is worth noting that the code works fine when there are no additions to the file contents. On adding a student to the file, the SORT does not work.

Original File contents

001,Joe,Bloggs,Test1:99,Test2:100,Test3:1
002,Ash,Smith,Test1:20,Test2:20,Test3:100
003,Jonathan,Peter,Test1:99,Test2:33,Test3:44

File Contents on adding a test student:

001,Joe,Bloggs,Test1:99,Test2:100,Test3:1
002,Ash,Smith,Test1:20,Test2:20,Test3:100
003,Jonathan,Peter,Test1:99,Test2:33,Test3:44
006,Mulch,Cart,Test1:99,Test2:22,Test3:11

The resultant error occurs at this stage (when the new student has been added). The sort function otherwise works fine.

Update and clarification:

For teaching purposes, I need it to work both on repl.it AND IDLE>

If someone could post a repl.it as an answer (With my code above, working), which also works when implemented in IDLE with a txt file, I will accept as an answer.

Answer 1

The reason of your issue here is that you're not appending to csv properly.

On Windows, the csv module has a bug/limitation when running Windows. It adds extra blank lines every line (actually it adds an extra carriage return char) if you don't open the file properly. So to fix it:

Python 3:

with open('studentinfo.txt','a',newline='') as fo:

Python 2:

with open('studentinfo.txt','ab') as fo:

So csv module is adding an extra \r at the end of your file. And when reading it back again, it issues an empty row.

It works fine in repl.it because they're using a python engine which runs on a Linux sandbox) , but the documentation still advises to open the files like I've shown.

(the documentation of the csv module is clear about this, even if it advises to do the same for read mode, and I never had any issue with a simple open("file.csv"))

Also see an old question of mine: portable way to write csv file in python 2 or python 3

If there's a double carriage return char at the end of the file, you don't see it (use Notepad++ with "show all symbols" to see a double CRCR character) but csv.reader returns an empty row which fails when sort uses your key function to compare it.

Now, if you want to be robust to that (because other people could edit your database, for instance, using excel csv mode or other terrible stuff):

I would filter out & sort & convert to list at the same time using filter(None,...) which removes "falsy" (i.e empty) rows:

sorted_list = sorted(filter(None,reader),key=lambda x: x[2])

Of course if a row has only 1 or 2 items, that will also fail. In that case, drop filter because we'd have to write a lambda and it's not worth, in favor of a generator comprehension:

sorted_list = sorted((x for x in reader if len(x)>2),key=lambda x: x[2])