ValueError: Tokenizer class MarianTokenizer does not exist or is not currently imported

Question

Get this error when trying to run a MarianMT-based nmt model.

Traceback (most recent call last):
File "/home/om/Desktop/Project/nmt-marionmt-api/inference.py", line 45, in <module>
    print(batch_inference(model_path="en-ar-model/Mark2", text=text))
  File "/home/om/Desktop/Project/nmt-marionmt-api/inference.py", line 15, in batch_inference
    tokenizer = AutoTokenizer.from_pretrained(model_path, local_file_only=True)
  File "/home/om/.virtualenvs/marianmt-api/lib/python3.8/site-packages/transformers/models/auto/tokenization_auto.py", line 525, in from_pretrained
    raise ValueError(
ValueError: Tokenizer class MarianTokenizer does not exist or is not currently imported.

Answer 1

You need to install the latest transformers-4.20.1.dev0. The reason for this problem is that the wrong version of the library is installed

Answer 2

Installing SentencePiece worked for me.

pip install sentencepiece