Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Keras LSTM input dimensions with one hot text embedding

I have 70k samples of text which I have embedded using Keras 'one hot' preprocessing. This gives me an array of [40, 20, 142...] which I then pad for a length of 28 (the longest sample length). All I am trying to do is predict these values to some categorical label (0 to 5 lets say). When I train the model I cannot get anything beyond -.13% accuracy (currently my error is this I have tried many ways to pass the input).

This is my data currently and am just trying to create a simple LSTM. Again my data is X -> [length of 28 integer values, embeddings] and Y -> [1 integer of length 3, (100, 143 etc.)]. Any idea what I am doing wrong?? I have asked many people and no one has been able to help. Here is the code for my model... any ideas? :(

optimizer = RMSprop(lr=0.01) #saw this online, no idea
model = Sequential()
model.add(Embedding(input_dim=28,output_dim=1,init='uniform')) #28 features, 1 dim output?
model.add(LSTM(150)) #just adding my LSTM nodes
model.add(Dense(1)) #since I want my output to be 1 integer value

model.compile(loss='sparse_categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
print(model.summary())

Edit:

using model.add(Embedding(input_dim=900,output_dim=8,init='uniform')) seems to work however still the accuracy never improves, I am at a loss of what to do.

like image 494
Maximus12793 Avatar asked Dec 21 '16 04:12

Maximus12793


People also ask

What is input_dim and output_dim in keras?

input_dim: Integer. Size of the vocabulary, i.e. maximum integer index + 1. output_dim: Integer. Dimension of the dense embedding. embeddings_initializer: Initializer for the embeddings matrix (see keras.initializers ).

What is input_length in keras?

It defines the size of the output vectors from this layer for each word. For example, it could be 32 or 100 or even larger. Test different values for your problem. input_length: This is the length of input sequences, as you would define for any input layer of a Keras model.

What are the different types of embeddings in keras?

There are a few different embedding vector sizes, including 50, 100, 200 and 300 dimensions. You can download this collection of embeddings and we can seed the Keras Embedding layer with weights from the pre-trained embedding for the words in your training dataset.

What is the use of the turn layer in keras?

Turns positive integers (indexes) into dense vectors of fixed size. This layer can only be used as the first layer in a model. input_dim: Integer. Size of the vocabulary, i.e. maximum integer index + 1. output_dim: Integer. Dimension of the dense embedding. embeddings_initializer: Initializer for the embeddings matrix (see keras.initializers ).


1 Answers

I have two suggestions.

  1. Use one hot representation for the target variable(y) also. If you give Y as integer, it will become a regression problem. Only if you give a one hot encoding, it becomes a classification problem.
  2. Try word2vec embedding when you have large amount of text, instead of one hot embedding.

optimizer = RMSprop(lr=0.01) 
embedding_vecor_length = 32
max_review_length = 28
nb_classes= 8
model = Sequential()
model.add(Embedding(input_dim=900, output_dim=embedding_vecor_length,
                    input_length=max_review_length)) 

model.add(LSTM(150))

#output_dim is a categorical variable with 8 classes
model.add(Dense(output_dim=nb_classes, activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
print(model.summary())

model.fit(X_train, y_train, nb_epoch=3, batch_size=64)

# Final evaluation of the model
scores = model.evaluate(X_test, y_test, verbose=0)
print("Accuracy: %.2f%%" % (scores[1]*100))

like image 120
Venkatachalam Avatar answered Oct 05 '22 06:10

Venkatachalam