Huggingface AlBert tokenizer NoneType error with Colab

Question

I simply tried the sample code from hugging face website: https://huggingface.co/albert-base-v2

`from transformers import AlbertTokenizer, AlbertModel` 
`tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')`
`text = "Replace me by any text you'd like."`
`encoded_input = tokenizer(text, return_tensors='pt')`

then I got the following error at the tokenizer step: ----> 5 encoded_input = tokenizer(text, return_tensors='pt')

TypeError: 'NoneType' object is not callable

I tried the same code on my local machine, it worked no problem. The problem seems within Colab. However, I do need help to run this model on colab GPU.

My python version on colab is Python 3.6.9

Answer 1

I found the answer. After install, import the AlbertTokenizer and Tokenizer=..., I received an error asking me to install 'SentencePiece package'. However, after I install this package and run tokenizer again, I started receiving the error above. So I open a brand new colab session, and install everything including the SentencePiece before creating tokenizer, and this time it worked. The Nonetype error simply means it doesn't know what is 'albert-base-v2'. However if you install the packages in right order colab will recognize better the relationship between AlbertTokenizer and SentencePiece. In short for this to work in colab 0. Open a new Colab session 1. Install Transformers and SentencePiece 2. import AlbertTokenizer 3.create tokenizer.

Answer 2

MeiNan Zhu's answer is correct.

Installing or importing SentencePiece before transformers works.

pip install Sentencepiece
!pip install transformers

tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased', do_lower_case=True)

type(tokenizer)

transformers.models.xlnet.tokenization_xlnet.XLNetTokenizer

Answer 3

I just tried restarting the kernel and run again after installing - !pip install sentencepiece - and it worked..