Tenserflow model for text classification doesn't predict as expected?

Question

I am trying to train a model for sentiment analysis and it shows an accuracy of 90% when splitting the data into training and testing! But whenever I am testing it on a new phrase is has pretty much the same result(usually it's in the range 0.86 - 0.95)! Here is the code:

sentences = data['text'].values.astype('U')
y = data['label'].values

sentences_train, sentences_test, y_train, y_test = train_test_split(sentences, y, test_size=0.2, random_state=1000)

tokenizer = Tokenizer(num_words=5000)
tokenizer.fit_on_texts(sentences_train)

X_train = tokenizer.texts_to_sequences(sentences_train)
X_test = tokenizer.texts_to_sequences(sentences_test)

vocab_size = len(tokenizer.word_index) + 1  

maxlen = 100

X_train = pad_sequences(X_train, padding='post', maxlen=maxlen)
X_test = pad_sequences(X_test, padding='post', maxlen=maxlen)

embedding_dim = 50
model = Sequential()
model.add(Embedding(input_dim=vocab_size, output_dim=embedding_dim,input_length=maxlen))
model.add(layers.Flatten())
model.add(layers.Dense(10, activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))
model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])
model.summary()

history = model.fit(X_train, y_train,
                    epochs=5,
                    verbose=True,
                    validation_data=(X_test, y_test),
                    batch_size=10)
loss, accuracy = model.evaluate(X_train, y_train, verbose=False)
print("Training Accuracy: {:.4f}".format(accuracy))
loss, accuracy = model.evaluate(X_test, y_test, verbose=False)
print("Testing Accuracy:  {:.4f}".format(accuracy))

The training data is a CSV file with 3 coumns: (id, text, labels(0,1)), where 0 is positive and 1 is negative.

Training Accuracy: 0.9855
Testing Accuracy:  0.9013

Testing it on new sentences like 'This is just a text!' and 'hate preachers!' would predict the same result [0.85],[0.83].

Answer 1

It seems that you're victim of overfitting. In other words, our model would overfit to the training data. Although it’s often possible to achieve high accuracy on the training set, as in your case, what we really want is to develop models that generalize well to testing data (or data they haven’t seen before).

You can follow these steps in order to prevent overfitting.

Also, in order to increase the algorithm performance I suggest you to increase the number of neurons for Dense layer and set more epochsin order to increase the performance of the algorithm when comes to testing it to new data.