Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How should I handle input data with nan values in TensorFlow?

Tags:

nan

tensorflow

I have a multilayer perceptron with a sigmoid loss (tf.nn.sigmoid_cross_entropy_with_logits) and an Adam optimizer (tf.train.AdamOptimizer). My input data has several features and some nan feature-values. When I replace the nan values with 0, I get a result, however, when I do not replace the nan values, I get loss=nan.

What is the best way to handle nan values in TensorFlow, and how can I use my input data with nan values without replacing them with 0?

like image 530
TomDriftwood Avatar asked Sep 19 '17 05:09

TomDriftwood


People also ask

What's the best way to handle NaN values?

Because most of the machine learning models that you want to use will provide an error if you pass NaN values into it. The easiest way is to just fill them up with 0, but this can reduce your model accuracy significantly.

How does dataset handle NaN values?

Pandas treat None and NaN as essentially interchangeable for indicating missing or null values. To facilitate this convention, there are several useful functions for detecting, removing, and replacing null values in Pandas DataFrame : isnull() notnull()

How do you deal with null values in a dataset?

There are 2 primary ways of handling missing values: Deleting the Missing values. Imputing the Missing Values.


1 Answers

Question

How can I somehow tell my network to ignore some input data. For example when the input data is nan

Answer

This is very similar to adding a mask to your input data. You want your input data to pass through, nans turned to zeros, but you want somehow to also signal to the neural network to ignore where the nans were and pay attention to everything else.

In this question about adding a mask I review how a mask can successfully be added to an image but also give a code demonstration for a non-image problem.

  • First create a mask, 1's where data exists in the input and 0's where nan exist.
  • Second, clean up the input converting nans to 0's, or 0.5's, or anything really.
  • Third, stack the mask onto the input. If the input is an image, then the mask becomes another colour channel.

The code in the masking question shows that when the mask is added the neural net is able to learn well and when the mask is not added it is not able to learn well.

like image 168
Anton Codes Avatar answered Oct 13 '22 21:10

Anton Codes