AttributeError: 'AcceleratorState' object has no attribute 'distributed_type'

Question

import transformers
from datasets import load_dataset
import tensorflow as tf

tokenizer = transformers.AutoTokenizer.from_pretrained('roberta-base')

df = load_dataset('csv', data_files={'train':'FinalDatasetTrain.csv', 'test':'FinalDatasetTest.csv'})

def tokenize_function(examples):
    return tokenizer(examples["text"], truncation=True)

tokenized_datasets = df.map(tokenize_function, batched=True)
data_collator = transformers.DataCollatorWithPadding(tokenizer=tokenizer)

model = transformers.AutoModelForSequenceClassification.from_pretrained('roberta-base', num_labels=7)

training_args = transformers.TFTrainingArguments(
    output_dir="./results",
    num_train_epochs=2,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=16,
    save_strategy='epoch',
    evaluation_strategy="epoch",
    logging_dir="./logs",
)

trainer = transformers.Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_datasets['train'],
    eval_dataset=tokenized_datasets['test'],
    data_collator=data_collator,
    tokenizer=tokenizer
)

trainer.train()

When I run this code I get an error saying:

AttributeError: 'AcceleratorState' object has no attribute 'distributed_type'.

How do I fix this (I tried both Jupyter notebook and Google Colab)?

Answer 1

To temporarily solve the problem, I downgrade the accelerate and transformers to:

• accelerate 0.15.0
• transformers 4.28.1
• tokenizers 0.13.3

Any major version higher than these will cause the error. I cannot find any official documentation about the change of model structure regarding distributed_type yet.

Remember to restart the runtime after any version change.

Note: downgrading is just a temp solution; I usually suggest upgrading to the latest version.

---

UPDATE

This is still valid in May 2024. The latest version of tokenizers and accelerate do not work:

• transformers 4.41.0
• accelerate 0.30.1
• tokenizers 0.19.1

which will still give the captioned error.

---

UPDATE 2 still not yet fixed in February 2025, with the following versions:

• transformers 4.49.0
• accelerate 1.4.0
• tokenizers 0.21.0

Same error occurs.