Does making DataFrame smaller makes it faster?

Question

I read an article (https://www.ritchieng.com/pandas-making-dataframe-smaller-faster/) which mentions that it makes the DataFrame faster by making it smaller (by converting data type).

Is there any association between smaller (memory usage) and faster (cpu time)?

Let's say I have a DataFrame column of int64. If we convert it to int8, will the operation on the column be faster? e.g. assuming that the operation is d[col] = d[col] + 1

Answer 1

why not to test it?

int64 dtype

In [29]: df = pd.DataFrame(np.random.randint(100, size=(10**7, 10), dtype="int64"))

In [30]: df.dtypes
Out[30]:
0    int64
1    int64
2    int64
3    int64
4    int64
5    int64
6    int64
7    int64
8    int64
9    int64
dtype: object

memory usage (in MiB):

In [31]: df.memory_usage().sum() / 1024**2
Out[31]: 762.9395294189453

timings:

In [32]: %timeit df.agg(["min","max","mean"])
4.68 s ± 84.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [33]: %timeit df+1
818 ms ± 26 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

---

int8 dtype

In [34]: df2 = df.astype("int8")

In [35]: df2.dtypes
Out[35]:
0    int8
1    int8
2    int8
3    int8
4    int8
5    int8
6    int8
7    int8
8    int8
9    int8
dtype: object

memory usage (in MiB):

In [38]: df2.memory_usage().sum() / 1024**2
Out[38]: 95.36750793457031

timings:

In [36]: %timeit df2.agg(["min","max","mean"])
2.4 s ± 222 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [37]: %timeit df2+1
170 ms ± 10.6 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

Answer 2

Changing the data type from int64 to int8 reduces the amount of bits required to store the data. This will drastically reduce your memory usage which will be very helpful when you have large data for intense computation. This increases the size of data that's possible with pandas before you get a memory error.

What will also increase performance is changing object-type columns to specific types since this allows type-optimisations.