Generating dictionaries to categorize tweets into pre-defined categories using NLTK

Question

I have a list of twitter users (screen_names) and I need to categorise them into 7 pre-defined categories - Education, Art, Sports, Business, Politics, Automobiles, Technology based on thier interest area. I have extracted last 100 tweets of the users in Python and created a corpus for each user after cleaning the tweets.

As mentioned here Tweet classification into multiple categories on (Unsupervised data/tweets) :
I am trying to generate dictionaries of common words under each category so that I can use it for classification.

Is there a method to generate these dictionaries for a custom set of words automatically?

Then I can use these for classifying the twitter data using a tf-idf classifier and get the degree of correspondence of the tweet to each of the categories. The highest value will give us the most probable category of the tweet.

But since the categorisation is based on these pre-generated dictionaries, I am looking for a way to generate them automatically for a custom list of categories.

Sample dictionaries :

Education - ['book','teacher','student'....]

Automobiles - ['car','auto','expo',....]

Example I/O:

**Input :** 
UserA - "students visited share learning experience eye opening 
article important preserve linaugural workshop students teachers 
others know coding like know alphabets vision driving codeindia office 
initiative get students tagging wrong people apologies apologies real 
people work..."
.
.
UserN - <another corpus of cleaned tweets>


**Expected output** : 
UserA - Education (61%)
UserN - Automobiles (43%)

Answer 1

TL;DR

Labels are necessary for supervised machine learning. And if you don't have training data that contains Xs (input texts) and Y (output labels) then (i) supervised learning might not be what you're looking for or (ii) you have to create a dataset with texts and their corresponding labels.

In Long

Lets try to break it down and see reflect what you're looking for.

I have a list twitter users (screen_names) and I need to categorise them into 7 pre-defined categories - Education, Art, Sports, Business, Politics, Automobiles, Technology

So your ultimate task is to label tweets into 7 categories.

I have extracted last 100 tweets of the users in Python and created a corpus for each user after cleaning the tweets.

100 data points is definitely insufficient to do anything if you want to train a supervised machine learning model from scratch.

Another thing is the definition of corpus. A corpus is a body of text so it's not wrong to call any list of strings a corpus. However, to do any supervised training, each text should come with the corresponding label(s)

But I see some people do unsupervised classification without any labels!

Now, that's an oxymoron =)

Unsupervised Classification

Yes, there are "unsupervised learning" which often means to learn representation of the inputs, generally the representation of the inpus is use to (i) generate or (ii) sample.

Generation from a representation means to create from the representation a data point that is similar to the data which an unsupervised model has learnt from. In the case of text process / NLP, this often means to generate new sentences from scratch, e.g. https://transformer.huggingface.co/

Sampling a representation means to give the unsupervised model a text and the model is expected to provide some signal from which the unsupervised model has learnt from. E.g. given a language model and novel sentence, we want to estimate the probability of the sentence, then we use this probability to compare across different sentences' probabilities.

Algorithmia has a nice summary blogpost https://algorithmia.com/blog/introduction-to-unsupervised-learning and a more modern perspective https://sites.google.com/view/berkeley-cs294-158-sp20/home

That's a whole lot of information but you don't tell me how to #$%^&-ing do unsupervised classification!

Yes, the oxymoron explanation isn't finished. If we look at text classification, what are we exactly doing?

We are fitting the input text into some pre-defined categories. In your case, the labels are pre-defined but

Q: Where exactly would the signal come from?

A: From the tweets, of course, stop distracting me! Tell me how to do classification!!!

Q: How do you tell the model that a tweet should be this label and not another label?

A: From the unsupervised learning, right? Isn't that what unsupervised learning supposed to do? To map the input texts to the output labels?

Precisely, that's the oxymoron,

Supervised learning maps the input texts to output labels not unsupervised learning

So what do I do? I need to use unsupervised learning and I want to do classification.

Then the question is ask is:

• Do you have labelled data?

  • If no, then how to get labels?

    • Use proxies, find signals that tells you a certain tweet is a certain label, e.g. from the hashtags or make some assumptions that some people always tweets on certain category
    • Use existing tweet classifiers to label your data and then train the classification model on the data
      • Do I have to pay for these classifiers? Most often, yes you do. https://english.api.rakuten.net/search/text%20classification

  • If yes, then how much?

    • If it's too little,
      • then how to create more? Maybe https://machinelearningmastery.com/a-gentle-introduction-to-the-bootstrap-method/
      • or maybe use some modern post-training algorithm https://towardsdatascience.com/https-medium-com-chaturangarajapakshe-text-classification-with-transformer-models-d370944b50ca

How about all these AI I keep hearing about, that I can do classification with 3 lines of code.

Don't they use unsupervised language models that sounds like Sesame Street characters, e.g. ELMO, BERT, ERNIE?

I guess you mean something like https://github.com/ThilinaRajapakse/simpletransformers#text-classification

from simpletransformers.classification import ClassificationModel
import pandas as pd


# Train and Evaluation data needs to be in a Pandas Dataframe of two columns. The first column is the text with type str, and the second column is the label with type int.
train_data = [['Example sentence belonging to class 1', 1], ['Example sentence belonging to class 0', 0]]
train_df = pd.DataFrame(train_data)

eval_data = [['Example eval sentence belonging to class 1', 1], ['Example eval sentence belonging to class 0', 0]]
eval_df = pd.DataFrame(eval_data)

# Create a ClassificationModel
model = ClassificationModel('bert', 'bert-base') # You can set class weights by using the optional weight argument

# Train the model
model.train_model(train_df)

Take careful notice of the comment:

Train and Evaluation data needs to be in a Pandas Dataframe of two columns. The first column is the text with type str, and the second column is the label with type int.

Yes that's the more modern approach to:

• First use a pre-trained language model to convert your texts into input representations
• Then feed the input representations and their corresponding labels to a classifier

Note, you still can't avoid the fact that you need labels to train the supervised classifier

Wait a minute, you mean all these AI I keep hearing about is not "unsupervised classification".

Genau. There's really no such thing as "unsupervised classification" (yet), somehow the (i) labels needs to be manually defined, (ii) the mapping between the inputs to the labels should exist

The right word to define the paradigm would be transfer learning, where the language is

• learned in a self-supervised manner (it's actually not truly unsupervised) so that the model learns to convert any text into some numerical representation

• then use the numerical representation with labelled data to produce the classifier.