How to train and tune an artificial multilayer perceptron neural network using Keras?

Question

I am building my first artificial multilayer perceptron neural network using Keras.

This is my input data:

This is my code which I used to build my initial model which basically follows the Keras example code:

model = Sequential()
model.add(Dense(64, input_dim=14, init='uniform'))
model.add(Activation('tanh'))
model.add(Dropout(0.5))
model.add(Dense(64, init='uniform'))
model.add(Activation('tanh'))
model.add(Dropout(0.5))
model.add(Dense(2, init='uniform'))
model.add(Activation('softmax'))

sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='mean_squared_error', optimizer=sgd)
model.fit(X_train, y_train, nb_epoch=20, batch_size=16)

Output:

Epoch 1/20
1213/1213 [==============================] - 0s - loss: 0.1760     
Epoch 2/20
1213/1213 [==============================] - 0s - loss: 0.1840     
Epoch 3/20
1213/1213 [==============================] - 0s - loss: 0.1816     
Epoch 4/20
1213/1213 [==============================] - 0s - loss: 0.1915     
Epoch 5/20
1213/1213 [==============================] - 0s - loss: 0.1928     
Epoch 6/20
1213/1213 [==============================] - 0s - loss: 0.1964     
Epoch 7/20
1213/1213 [==============================] - 0s - loss: 0.1948     
Epoch 8/20
1213/1213 [==============================] - 0s - loss: 0.1971     
Epoch 9/20
1213/1213 [==============================] - 0s - loss: 0.1899     
Epoch 10/20
1213/1213 [==============================] - 0s - loss: 0.1957     
Epoch 11/20
1213/1213 [==============================] - 0s - loss: 0.1923     
Epoch 12/20
1213/1213 [==============================] - 0s - loss: 0.1910     
Epoch 13/20
1213/1213 [==============================] - 0s - loss: 0.2104     
Epoch 14/20
1213/1213 [==============================] - 0s - loss: 0.1976     
Epoch 15/20
1213/1213 [==============================] - 0s - loss: 0.1979     
Epoch 16/20
1213/1213 [==============================] - 0s - loss: 0.2036     
Epoch 17/20
1213/1213 [==============================] - 0s - loss: 0.2019     
Epoch 18/20
1213/1213 [==============================] - 0s - loss: 0.1978     
Epoch 19/20
1213/1213 [==============================] - 0s - loss: 0.1954     
Epoch 20/20
1213/1213 [==============================] - 0s - loss: 0.1949

How do I train and tune this model and get my code to output my best predictive model? I am new to neural networks and am just wholly confused as to what is the next step after building the model. I know I want to optimize it, but I'm not sure which features to tweak or if I am supposed to do it manually or how to write code to do so.

Answer 1

Some things that you could do are:

• Change your loss function from mean_squared_error to binary_crossentropy. mean_squared_error is intended for regression, but you want to classify your data.
• Add show_accuracy=True to your fit() function, which outputs the accuracy of your model at every epoch. That information is probably more useful to you than just the loss value.
• Add validation_split=0.2 to your fit() function. Currently you are only training on a training set and validating on nothing. That's a no-go in machine learning as you can't be sure that your model hasn't simply memorized the correct answers for your dataset (without really understanding why these answers are correct).
• Change from Obama/Romney to Democrat/Republican and add data from previous elections. ~1200 examples is a pretty small dataset for neural networks. Also add columns with valuable information, like unemployment rate or population density. Note that quite some of the values (like population number) are probably similar to providing the name of the state, so e.g. your net will likely learn that Texas means Republican.
• If you haven't done that already, normalize all your values to the range of 0 to 1 (by subtracting from each value the minimum of the column and then dividing by the (max - min) of the column). Neural networks can handle normalized data better than unnormalized data.
• Try Adam and Adagrad instead of SGD. Sometimes they perform better. (See documentation about optimizers.)
• Try Activation('relu'), LeakyReLU, PReLU and ELU instead of Activation('tanh'). Tanh is rarely the best choice. (See advanced activation functions.)
• Try increasing/decreasing your dense layers sizes (e.g. from 64 to 128). Also try adding/removing layers.
• Try adding BatchNormalization layers (before the Activation layers). (See documentation.)
• Try changing the dropout rates (e.g. from 0.5 to 0.25).