Cycling a binary file in Python - Remove from beginning add to end

Question

Could anyone point me towards a method of cycling a binary file in Python? I have a file full of 4 byte integers basically and when the file reaches a certain size, i.e. a certain number of values have been written, I want to start removing one from the start and adding one at the end.

I'm still reasonably new to Python, so just trying to think of a neat way of doing this.

Thanks.

Answer 1

My idea: the first integer in the file gives you the position of the actual beginning of the data. At the start this will be 4 (assuming an integer takes 4 bytes). When the file is full, you just start overwriting data at the beginning and increase the position integer. This is basically a simple ring-buffer in file-form.

Answer 2

2000 numbers?

That's 16K. Do it in memory. Indeed, by declaring your buffers to be 16K, you can probably do the entire operation in a single I/O request. And on some large 64-bit systems, 2000 numbers more-or-less is the default buffer size.

Your data volume is microscopic. Don't waste time optimizing such a minuscule amount of data.

with open( "my file.dat", "rb", 16384 ) as the_file:
    my_circular_queue = list( read_the_numbers( the_file ) )

if len(my_circular_queue) >=  2000:
    my_circular_queue = my_circular_queue[1:]
my_circular_queue.append( a_new_number )

with open( "my file.dat", "wb", 16384 ) as the_file:
    write_the_numbers( the_file, my_circular_queue )

It totally fits in memory. Don't waste time trying to finesse a complex update.