Menu
I want to train gpt 2 from scratch but there is only fine-tuning approach based on pretrained models in articles I found. I've used this https://github.com/nshepperd/gpt-2 for train with existing model. Should I edit these Python scripts to train from scratch?
508
There is a project called Teachable NLP that you can train GPT-2 Model with your own text. It is very easy to use. You just simply upload your text file and it will train GPT-2 model automatically.
Both are unsupervised transformer models trained to generate text by predicting the next word in a sequence of tokens. The GPT-2 model has 1.5 billion parameters, and was trained on a dataset of 8 million web pages.
GPT-2 was known to have poor performance when given tasks in specialized areas such as music and storytelling. GPT-3 can now go further with tasks such as answering questions, writing essays, text summarization, language translation, and generating computer code.
I found the answer in 'issues' of this repo https://github.com/nshepperd/gpt-2
If you want to not use the released model at all, for instance because you want to train a model with incompatible hyperparameters, it should be sufficient to just skip the restore from the released model checkpoint (around train.py:164-177) on your first run so the parameters will all be randomly initialized.
174
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
