How to read specific part of large file in Python

Question

Given a large file (hundreds of MB) how would I use Python to quickly read the content between a specific start and end index within the file?

Essentially, I'm looking for a more efficient way of doing:

open(filename).read()[start_index:end_index]

Answer 1

You can seek into the file the file and then read a certain amount from there. Seek allows you to get to a specific offset within a file, and then you can limit your read to only the number of bytes in that range.

with open(filename) as fin:
    fin.seek(start_index)
    data = fin.read(end_index - start_index)

That will only read that data that you're looking for.

Answer 2

This is my solution with variable width encoding. My CSV file contains a dictionary where each row is a new item.

def get_stuff(filename, count, start_index):
    with open(filename, 'r') as infile:
             reader = csv.reader(infile)
             num = 0 
             for idx, row in enumerate(reader):
                 if idx >= start_index-1:
                     if num >= count:
                         return
                 else:
                     yield row 
                     num += 1