Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Find NA values after using addNA()

Tags:

r

na

categories

I have a data frame with a bunch of categorical variables. Some of them contain NA's and I use the addNA function to convert them to an explicit factor level. My problem comes when I try to treat them as NA's they don't seem to register.

Here's my example data set and attempts to 'find' NA's:

df1 <- data.frame(id = 1:200, y =rbinom(200, 1, .5),
                  var1 = factor(rep(c('abc','def','ghi','jkl'),50)))
df1$var2 <- factor(rep(c('ab c','ghi','jkl','def'),50))
df1$var3 <- factor(rep(c('abc','ghi','nop','xyz'),50))

df1[df1$var1 == 'abc','var1'] <- NA

df1$var1 <- addNA(df1$var1)

df1$isNaCol <- ifelse(df1$var1 == NA, 1, 0);summary(df1$isNaCol)
df1$isNaCol <- ifelse(is.na(df1$var1), 1, 0);summary(df1$isNaCol)
df1$isNaCol <- ifelse(df1$var1 == 'NA', 1, 0);summary(df1$isNaCol)
df1$isNaCol <- ifelse(df1$var1 == '<NA>', 1, 0);summary(df1$isNaCol)

Also when I type ??addNA I don't get any matches. Is this a gray-market function or something? Any suggestions would be appreciated.

like image 313
screechOwl Avatar asked Jun 25 '13 15:06

screechOwl


People also ask

How do I check for NA values in R?

To check which value in NA in an R data frame, we can use apply function along with is.na function. This will return the data frame in logical form with TRUE and FALSE.

How do you get the NA factor?

You can use addNA() . addNA modifies a factor by turning NA into an extra level (so that NA values are counted in tables, for instance). So another reason this is nice is because if you already have a factor f , you can use addNA() to quickly add NA as a factor level without changing f .

How do you get NA rows in R?

The following in-built functions in R collectively can be used to find the rows and column pairs with NA values in the data frame. The is.na() function returns a logical vector of True and False values to indicate which of the corresponding elements are NA or not.

How do I select columns with NA in R?

There are two easy methods to select columns of an R data frame without missing values, first one results in a vector and other returns a matrix. For example, if we have a data frame called df then the first method can be used as df[,colSums(is.na(df))==0] and the second method will be used as t(na. omit(t(df))).


3 Answers

Testing equality to NA with the usual comparison operators always yields NA---you want is.na. Additionally, calling is.na on a factor test each level index (not the value associated with that index), so you want to convert the factor to a character vector first.

df1$isNaCol <- ifelse(is.na(as.character(df1$var1)), 1, 0);summary(df1$isNaCol)
like image 133
Matthew Plourde Avatar answered Oct 06 '22 01:10

Matthew Plourde


Note that this is done with the OP's data before the call to addNA().

It is instructive to see what addNA() does with this data.

> head(df1$var1)
[1] <NA> def  ghi  jkl  <NA> def 
Levels: abc def ghi jkl
> levels(df1$var1)
[1] "abc" "def" "ghi" "jkl"
> head(addNA(df1$var1))
[1] <NA> def  ghi  jkl  <NA> def 
Levels: abc def ghi jkl <NA>
> levels(addNA(df1$var1))
[1] "abc" "def" "ghi" "jkl" NA

addNA is altering the levels of the factor such that missing-ness (NA) is a level where by default R ignores it as what level the NA values take is, of course, missing. It is also stripping out the NA information - in a sense it is no longer unknown but part of a category "missing".

To look at the help for addNA us ?addNA.

If we look at the definition of addNA we see that all it is doing is altering the levels

of the factor, not changing the data any:

> addNA
function (x, ifany = FALSE) 
{
    if (!is.factor(x)) 
        x <- factor(x)
    if (ifany & !any(is.na(x))) 
        return(x)
    ll <- levels(x)
    if (!any(is.na(ll))) 
        ll <- c(ll, NA)
    factor(x, levels = ll, exclude = NULL)
}

Note that it doesn't otherwise change the data - the NA are still there in the factor. We can replicate most of the behaviour of addNA via:

with(df1, factor(var1, levels = c(levels(var1), NA), exclude = NULL))

> head(with(df1, factor(var1, levels = c(levels(var1), NA), exclude = NULL)))
[1] <NA> def  ghi  jkl  <NA> def 
Levels: abc def ghi jkl <NA>

However because NA is now a level, those entries are not indicated as being missing via is.na() That explains the second comparison you do not working (where you use is.na()).

The only nicety you get from addNA is that it doesn't add NA as a level if it already exists as one. Also, via the ifany you can stop it adding NA as a level if there are no NAs in the data.

Where you are going wrong is attempting to compare an NA with something using the usual comparison methods (except your second example). If we don't know what value and NA observation takes, how can we compare it with something? Well, we can't, other than with the internal representation of NA. This is what is done by the is.na() function:

> with(df1, head(is.na(var1), 10))
 [1]  TRUE FALSE FALSE FALSE  TRUE FALSE FALSE FALSE  TRUE FALSE

Hence I would do (without using addNA at all)

df1 <- transform(df1, isNaCol = is.na(var1))

> head(df1)
  id y var1 var2 var3 isNaCol
1  1 1 <NA> ab c  abc    TRUE
2  2 0  def  ghi  ghi   FALSE
3  3 0  ghi  jkl  nop   FALSE
4  4 0  jkl  def  xyz   FALSE
5  5 0 <NA> ab c  abc    TRUE
6  6 1  def  ghi  ghi   FALSE

If you want that as a 1, 0, variable, just add as.numeric() as in

df1 <- transform(df1, isNaCol = as.numeric(is.na(var1)))

Where I think you are really going wrong is in wanting to attach an NA level to the factor. I see addNA() as a convenience function for use in things like table(), and even that has arguments to not need the prior use of addNA(), e.g.:

> with(df1, table(var1, useNA = "ifany"))
var1
 abc  def  ghi  jkl <NA> 
   0   50   50   50   50
like image 35
Gavin Simpson Avatar answered Oct 06 '22 00:10

Gavin Simpson


Anything compared to NA is NA; this is why your first summary is all NA.

The addNA function changes any NA observations in your factor to a new level. This level is then given the label NA (of character mode). The underlying variable itself no longer has any NAs. This is why your second summary is all 0.

To see how many observations have the NA level, use what Matthew Plourde posted.

like image 44
Hong Ooi Avatar answered Oct 06 '22 01:10

Hong Ooi