I've looking for a solution to solve the slow upload speed of images dataset on google colab when i use a connection from GoogleDrive. Using the follow code:
from google.colab import drive
drive.mount('/content/gdrive')
Using this procedure i can upload images and create labels using a my def load_dataset
:
'train_path=content/gdrive/MyDrive/Capstone/Enviroment/cell_images/train'
train_files, train_targets = load_dataset(train_path)
But, as i said, it's very slow, especially because my full dataset is composed by 27560 images.
To solve my problem, i've tried to use this solution.
But now, in order to still use my def
function, after download the .tar
file i wanna extract in a specific folder in the colab enviroment. I found this answer but not solve my problem.
Example:
This is the environment with the test.tar already downloaded.
But i wanna extract the files in the tar file, which structure is train/Uninfected
; train/Parasitized
, to get this:
content
To use the path in def function:
train_path = train_path=content/cell_images/train/'
train_files, train_targets = load_dataset(train_path)
test_path = train_path=content/cell_images/test/'
test_files, test_targets = load_dataset(test_path)
valid_path = train_path=content/cell_images/valid/'
valid_files, valid_targets = load_dataset(valid_path)
I tried to use:
! mkdir -p content/cell_images
and
!tar -xvf 'test.tar' content/cell_images
But it doesn't work.
Does anyone know how to proceed?
Thanks!
To download the compressed file (or any file in general), you can use the ! wget command as below. Then, you will need to unzip the compressed file to open the files contained in it. ! unzip command will work for most files with extension .
How to Download Folders from Colab. The command is !zip followed by r which means “recursive”, then we write the file path of the zipped file (i.e. /content/sample_data. zip ) and finally, we write the folder that we want to zip (i.e. /content/sample_data ) and voila, the zip file is generated :-).
Although late answer, but might help others:
shutil.unpack_archive works with almost all archive formats (e.g., “zip”, “tar”, “gztar”, “bztar”, “xztar”) and it's simple:
import shutil
shutil.unpack_archive("filename", "path_to_extract")
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With