Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is there a way to remove nan from a dictionary filled with data?

I have a dictionary that is filled with data from two files I imported, but some of the data comes out as nan. How do I remove the pieces of data with nan?

My code is:

import matplotlib.pyplot as plt 
from pandas.lib import Timestamp
import numpy as np   
from datetime import datetime
import pandas as pd
import collections

orangebook = pd.read_csv('C:\Users\WEGWEIS_JAKE\Desktop\Work Programs\Code Files\products2.txt',sep='~', parse_dates=['Approval_Date'])
specificdrugs=pd.read_csv('C:\Users\WEGWEIS_JAKE\Desktop\Work Programs\Code Files\Drugs.txt',sep=',')

"""This is a dictionary that collects data from the .txt file
This dictionary has a key,value pair for every generic name with its corresponding approval date """
drugdict={}
for d in specificdrugs['Generic Name']:
    drugdict.dropna()
    drugdict[d]=orangebook[orangebook.Ingredient==d.upper()]['Approval_Date'].min()

What should I add or take away from this code to make sure that there are no key,value pairs in the dictionary with a value of nan?

like image 474
Jwegs32 Avatar asked Jun 05 '14 19:06

Jwegs32


People also ask

How do you exclude NaN values from a list?

To remove NaN from a list using Python, the easiest way is to use the isnan() function from the Python math module and list comprehension. You can also use the Python filter() function. The Python numpy module also provides an isnan() function that we can use to check if a value is NaN.

How do you remove a value from a dictionary?

To delete a key, value pair in a dictionary, you can use the del method. A disadvantage is that it gives KeyError if you try to delete a nonexistent key. So, instead of the del statement you can use the pop method.


3 Answers

from math import isnan

if nans are being stored as keys:

# functional
clean_dict = filter(lambda k: not isnan(k), my_dict)

# dict comprehension
clean_dict = {k: my_dict[k] for k in my_dict if not isnan(k)}

if nans are being stored as values:

# functional
clean_dict = filter(lambda k: not isnan(my_dict[k]), my_dict)

# dict comprehension
clean_dict = {k: my_dict[k] for k in my_dict if not isnan(my_dict[k])}
like image 112
twinlakes Avatar answered Nov 15 '22 21:11

twinlakes


With simplejson

import simplejson

clean_dict  = simplejson.loads(simplejson.dumps(my_dict, ignore_nan=True))
## or depending on your needs
clean_dict  = simplejson.loads(simplejson.dumps(my_dict, allow_nan=False))
like image 28
hangc Avatar answered Nov 15 '22 22:11

hangc


Instead of trying to remove the NaNs from your dictionary, you should further investigate why NaNs are getting there in the first place.

It gets difficult to use NaNs in a dictionary, as a NaN does not equal itself.

Check this out for more information: NaNs as key in dictionaries

like image 36
Greg Hilston Avatar answered Nov 15 '22 21:11

Greg Hilston