Menu
I was trying to run deep learning models using the cifar10 data set. When keras is installed on the system and one tries to use the data set, keras downloads it once in:
~/.keras/datasets/
I say that because I can ls that directory and I see:
$ ls
cifar-10-batches-py cifar-10-batches-py.tar.gz cifar-100-python cifar-100-python.tar.gz mnist.pkl.gz
however when I install keras in a docker container ~/.keras doesn't exist. I am sure its being saved somewhere because I can later load the data set when I log in to the docker image through a container with bash (with docker run -it --rm tf_img bash). However, I can't find here the files might be saved. Where are they saved?
Also once I find this location, I was planning to volume my local ~/.keras/ (or whatever computer I'm logged into) with the containers, so that the data sets don't have to be donwloaded 30 times every time I try to train some model.
First progress I found where .keras is using the find -type d -name .keras command in the container (its in ./root/.keras). However, it seems that even though I volumed the folder correctly it downloads it anyway (even though I've went to the location /root/.keras in the container and see the stuff there as it is in my local computer). I did:
docker run -it --rm -v /Users/folder1:/Users/folder1 -v /Users/Pinocchio/.keras:/root/keras tf_cpu cifar10_cnn.py
and it still downloads the data set. Why?
Another thing that strikes me a super odd is that when I bash into the container it starts me inside of root which it has never done that before.
857
Each container will have its own file system, optionally created with “volume mounts” that bind data from the host to the container. Images store the entire contents of the image on your drive. Whenever you pull an image from the internet, it’s downloaded and stored, usually forever.
In addition to providing many of the building blocks for neural networks, Keras also has many built-in datasets. In this post, we will take a look at some of the benchmark data from the Keras library. These data sets are often used to demonstrate improved performance, over state of the art methods, of novel machine learning algorithms.
They are stored in the part of the host filesystem managed specifically by Docker and it should not be modified by non-Docker processes. Volumes are the most preferred way to store container data as they provide efficient performance and are isolated from the other functionalities of the Docker host.
If you create a new volume, Docker will automatically create a new storage layer on the underlying host filesystem and initialize it with the data you specify. Once you have created a volume, you can use it to store any type of data, including static files, application data, or logs. You can also use volumes to share data between containers.
Seems like I solved it. To find where .keras is in the container do once logged in to the container with bash ( as in docker run -it --rm -v /local:/container image_name bash for example) :
find -type d -name .keras
to discover that the location is
./root/.keras
Now just volume the location of the data set:
docker run -it --rm -v /Users/folder1:/Users/folder1 -v /Users/Pinocchio/.keras:/root/.keras tf_cpu cifar10_cnn.py
make sure to not have any typos, specially since its .keras and not keras.
95
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With