Most efficient way to store list of integers

Question

I have recently been doing a project in which one of the aims is to use as little memory as possible to store a series of files using Python 3. Almost all of the files take up very little space, apart from one list of integers that is roughly 333,000 integers long and has integers up to about 8000 in size.

I'm currently using pickle to store the list, which takes up around 7mb, but I feel like there must be a more memory efficient way to do this.

I have tried storing it as a text file and csv, bur both of these used in excess of 10mb of space.

Answer 1

One stdlib solution you could use is arrays from array, from the docs:

This module defines an object type which can compactly represent an array of basic values: characters, integers, floating point numbers. Arrays are sequence types and behave very much like lists, except that the type of objects stored in them is constrained.

This generally sheds a bit of memory of large lists, for example, with a 10 million element a list, the array trims up 11mb:

import pickle    
from array import array

l = [i for i in range(10000000)]
a = array('i', l)

# tofile can also be used.
with open('arrfile', 'wb') as f:  
    pickle.dump(a, f)

with open('lstfile', 'wb') as f:
    pickle.dump(l, f)

Sizes:

!du -sh ./*
39M     arrfile
48M     lstfile