very low accuracy in Neural networks using keras and validation accuracy 0.0000e+00

Question

Below is the code which I am using. Please let me know why I get so low validation and training accuracy? Validation accuracy is just 0.0000e+00 and also training accuracy is approximately 37%. What could have possibly gone wrong? My training set has 10500 rows and 172 columns My test set has 3150 rows and 172 columns My first column is the response (class) and hence i use it only as Y and the rest columns as X. My response is 3 classes : default,LF and RF

from __future__ import print_function
import numpy as np
import pandas
from keras.models import Sequential
from keras.layers.core import Dense, Activation
from keras.optimizers import SGD
from keras.utils import np_utils
from sklearn.preprocessing import LabelEncoder
np.random.seed(1671)
NB_EPOCH = 5
BATCH_SIZE = 128
VERBOSE = 1
NB_CLASSES = 3
OPTIMIZER = SGD()
N_HIDDEN = 128
VALIDATION_SPLIT=0.1
RESHAPED = 171
dataframe_train = pandas.read_csv("TrainingEdgesToAction.csv", header=None)
dataset_train = dataframe_train.values
X_train = dataset_train[1:,1:172].astype(float)
#X_train = dataset_train[1:,0:172]
Y_train = dataset_train[1:,0]

dataframe_test = pandas.read_csv("TestingEdgesToAction.csv", header=None)
dataset_test = dataframe_test.values
X_test = dataset_test[1:,1:172].astype(float)
#X_test = dataset_test[1:,0:172]
Y_test = dataset_test[1:,0]

X_train = X_train.reshape(10500,RESHAPED)
X_test = X_test.reshape(3150,RESHAPED)
X_train /= 255
X_test /= 255
print(X_train.shape[0],'train samples')
print(X_test.shape[0],'test samples')

encoder = LabelEncoder()
encoder.fit(Y_train)
encoded_Y_train = encoder.transform(Y_train)
# convert integers to dummy variables (i.e. one hot encoded)
dummy_y_train = np_utils.to_categorical(encoded_Y_train)
print(dummy_y_train)

encoder = LabelEncoder()
encoder.fit(Y_test)
encoded_Y_test = encoder.transform(Y_test)
# convert integers to dummy variables (i.e. one hot encoded)
dummy_y_test = np_utils.to_categorical(encoded_Y_test)
print(dummy_y_test)

#Y_train = np_utils.to_categorical(Y_train,NB_CLASSES)
#Y_test = np_utils.to_categorical(Y_test, NB_CLASSES)

model = Sequential()
model.add(Dense(N_HIDDEN,input_shape=(RESHAPED,)))
model.add(Activation('relu'))
model.add(Dense(N_HIDDEN))
model.add(Activation('relu'))
model.add(Dense(NB_CLASSES))
model.add(Activation('softmax'))
model.summary()
model.compile(loss='categorical_crossentropy',optimizer=OPTIMIZER,metrics=
['accuracy'])
history = model.fit(X_train,dummy_y_train,batch_size=BATCH_SIZE,epochs=NB_EPOCH,shuffle=True,verbose=VERBOSE,validation_split=VALIDATION_SPLIT)
score = model.evaluate(X_test,dummy_y_test,verbose=VERBOSE)

print("\nTest score:",score[0])
print("Test accuracy:",score[1])

10500 train samples
3150 test samples
[[ 1.  0.  0.]
[ 1.  0.  0.]
[ 1.  0.  0.]
..., 
[ 0.  0.  1.]
[ 0.  0.  1.]
[ 0.  0.  1.]]
[[ 1.  0.  0.]
[ 1.  0.  0.]
[ 1.  0.  0.]
..., 
[ 0.  0.  1.]
[ 0.  0.  1.]
[ 0.  0.  1.]]
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_49 (Dense)             (None, 128)               22016     
_________________________________________________________________
activation_49 (Activation)   (None, 128)               0         
_________________________________________________________________
dense_50 (Dense)             (None, 128)               16512     
_________________________________________________________________
activation_50 (Activation)   (None, 128)               0         
_________________________________________________________________
dense_51 (Dense)             (None, 3)                 387       
_________________________________________________________________
activation_51 (Activation)   (None, 3)                 0         
=================================================================
Total params: 38,915
Trainable params: 38,915
Non-trainable params: 0
_________________________________________________________________
Train on 9450 samples, validate on 1050 samples
Epoch 1/5
9450/9450 [==============================] - 2s - loss: 1.0944 - acc: 0.3618 
- val_loss: 1.1809 - val_acc: 0.0000e+00
Epoch 2/5
9450/9450 [==============================] - 1s - loss: 1.0895 - acc: 0.3704 
- val_loss: 1.2344 - val_acc: 0.0000e+00
Epoch 3/5
9450/9450 [==============================] - 0s - loss: 1.0874 - acc: 0.3704 
- val_loss: 1.2706 - val_acc: 0.0000e+00
Epoch 4/5
9450/9450 [==============================] - 0s - loss: 1.0864 - acc: 0.3878 
- val_loss: 1.2955 - val_acc: 0.0000e+00
Epoch 5/5
9450/9450 [==============================] - 0s - loss: 1.0860 - acc: 0.3761 
- val_loss: 1.3119 - val_acc: 0.0000e+00
2848/3150 [==========================>...] - ETA: 0s
Test score: 1.10844093784
Test accuracy: 0.333333333333

Answer 1

I decided to summarize our "chat".

So, what to do if your test accuracy is low (around ≈ 0.1%), here are some general recommendations:

• Try different optimizers, based on my experience Adam is a good starting point.
• Try different activation functions; I recommend you to start with "relu", and try "selu" and "elu" as well.
• Add regularization. Dropout and BatchNormalization may increase your test accuracy.
• Give your network some time, train it longer.
• Play around with hyperparameters such as a number of layers, batch size, a number of epochs, learning rate and so on...
• Lastly, always normalize your data before feeding it to the NN.

Answer 2

so for me, I was looking for a metric to see how my regression model performed and for me it worked best with using R² as my main metric. R² can describe how “good” a model is at making predictions. As @Paddy already mentioned you need to give the model some time. At least 30 epochs in your case. So now when:

R² = 0 means the model always fails to predict the correct target variable R² = 1 means the model predicts the target variable perfectly.

In code it would look like this:

def det_coeff(y_true, y_pred):
u = K.sum(K.square(y_true - y_pred))
v = K.sum(K.square(y_true - K.mean(y_true)))
return K.ones_like(v) - (u / v)

that is the function for determining the metric.

model.compile(optimizer="SGD", loss="mse", metrics=[det_coeff])