tcmalloc: large alloc ... killed in Google Colab

Question

I was trying to setup MuseGAN in Google Colab. I already downloaded the data and I am now processing the data by storing it into the shared memory using the SharedArray package, by running the script ./scripts/process_data.sh. I encountered this error

> Loading data from '/content/musegan/scripts/../data/train_x_lpd_5_phr.npz'.
Saving data to shared memory.
tcmalloc: large alloc 6245990400 bytes == 0x26b6000 @  0x7f97d2bea1e7 0x7f97d08e0a41 0x7f97d0943bb3 0x7f97d08e4937 0x5553b5 0x5a730c 0x503073 0x507641 0x504c28 0x502540 0x502f3d 0x507641 0x501945 0x591461 0x59ebbe 0x545068 0x506b39 0x502209 0x502f3d 0x506859 0x504c28 0x506393 0x634d52 0x634e0a 0x6385c8 0x63915a 0x4a6f10 0x7f97d27e7b97 0x5afa0a
./scripts/process_data.sh: line 5:   360 Killed                  python "$DIR/../src/process_data.py" "$DIR/../data/train_x_lpd_5_phr.npz"

Can someone explain this? I don't get why I encountered this. I encountered this at first when I run it on a machine wihout a gpu (i.e. cpu only) then I heard about Google Colab.

Answer 1

This is a problem about how Google deals with increasing memory usage and thinks an OOM will occur even though it won't.

See https://github.com/huggingface/transformers/issues/4668