Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Where does hugging face's transformers save models?

Running the below code downloads a model - does anyone know what folder it downloads it to?

!pip install -q transformers
from transformers import pipeline
model = pipeline('fill-mask')
like image 763
user3472360 Avatar asked May 14 '20 13:05

user3472360


People also ask

Where are hugging face models saved?

Push your model from Python It will store your access token in the Hugging Face cache folder (by default ~/. cache/ ).

How do you save the Transformers model?

First way is to store a model like you have stored torch. save(model. state_dict(), PATH) and to load the same model on a different machine or some different place then first you have to make the instance of that model and then assign that model to the model parameter like this.

How do you contribute to a Huggingface model?

We recommend putting them in the following dataset: huggingface/documentation-images. If an external contribution, feel free to add the images to your PR and ask a Hugging Face member to migrate your images to this dataset.


Video Answer


4 Answers

Update 2021-03-11: The cache location has now changed, and is located in ~/.cache/huggingface/transformers, as it is also detailed in the answer by @victorx.


This post should shed some light on it (plus some investigation of my own, since it is already a bit older).

As mentioned, the default location in a Linux system is ~/.cache/torch/transformers/ (I'm using transformers v 2.7, currently, but it is unlikely to change anytime soon.). The cryptic folder names in this directory seemingly correspond to the Amazon S3 hashes.

Also note that the pipeline tasks are just a "rerouting" to other models. To know which one you are currently loading, see here. For your specific model, pipeline(fill-mask) actually utilizes a distillroberta-base model.

like image 97
dennlinger Avatar answered Oct 10 '22 14:10

dennlinger


As of Transformers version 4.3, the cache location has been changed.

The exact place is defined in this code section ​https://github.com/huggingface/transformers/blob/master/src/transformers/file_utils.py#L181-L187

On Linux, it is at ~/.cache/huggingface/transformers.

The file names there are basically SHA hashes of the original URLs from which the files are downloaded. The corresponding json files can help you figure out what are the original file names.

like image 13
victorx Avatar answered Oct 10 '22 16:10

victorx


On windows 10, replace ~ with C:\Users\username or in cmd do cd /d "%HOMEDRIVE%%HOMEPATH%".

So full path will be: C:\Users\username\.cache\huggingface\transformers

like image 7
Maverick Meerkat Avatar answered Oct 10 '22 14:10

Maverick Meerkat


As of transformers 4.22, the path appears to be (tested on CentOS):

~/.cache/huggingface/hub/
like image 2
Victor Yan Avatar answered Oct 10 '22 16:10

Victor Yan