Menu
I am trying to predict a continuous value (using a Neural Network for the first time). I have normalized the input data. I can't figure out why I am getting a loss: nan output starting with the first epoch.
I read and tried many suggestions from previous answers to the same question but that none of them helped me. My training data shape is: (201917, 64). Here's my code:
model = Sequential()
model.add(Dense(100, input_dim=X.shape[1], activation='relu'))
model.add(Dense(100, activation='relu'))
model.add(Dense(100, activation='relu'))
# Output layer
model.add(Dense(1, activation='linear'))
# Construct the neural network inside of TensorFlow
model.compile(loss='mean_squared_error', optimizer='Adam')
# train the model
model.fit(X_train, y_train, epochs=10, batch_size=32,
shuffle=True, verbose=2)
632
Faulty Loss function Reason: Sometimes the computations of the loss in the loss layers causes nans to appear. For example, Feeding InfogainLoss layer with non-normalized values, using custom loss layer with bugs, etc.
If you add loss every time you loop through the data in the batch, it will most probably lead to nan, instead, you need to calculate your loss by adding it every time a batch is forwarded through the network.
The constant learning rate is the default schedule in all Keras Optimizers. For example, in the SGD optimizer, the learning rate defaults to 0.01 . To use a custom learning rate, simply instantiate an SGD optimizer and pass the argument learning_rate=0.01 .
These are the steps that you can take to find the cause of your problem:
Make sure that your dataset is what it should be:
Normalize your model using Dropout, BatchNormalization, L1/L2 regularization, change your batch_size, or scale your data to other ranges (e.g. [-1, 1]).
Reduce the size of your network.
Change other hyper-parameters (e.g. optimizer or activation function).
You can check this and this link to get extra help.
185
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
