Reducing number of factor levels before modelling

Question

I have a factor with 2600 levels and I want to reduce it to ~10 before modelling

I thought I could do this with an operation that says "if a factor is listed fewer than x times, it should be placed into a bucket called "other"

Here is some example data:

df <- data.frame(colour=c("blue","blue","blue","green","green","orange","grey"))

And this is the output I am hoping for:

  colour
1   blue
2   blue
3   blue
4  green
5  green
6  other
7  other

I have tried the below:

df %>% mutate(colour = ifelse(count(colour) < 2, 'other', colour))

Error in mutate_impl(.data, dots) : Evaluation error: no applicable method for 'groups' applied to an object of class "factor".

Answer 1

There is actually a nice package in the tidyverse called forcats which helps in dealing with factors. You can use fct_lump, which does exactly what you need:

library(tidyverse)

df %>% mutate(colour = fct_lump(colour, n = 2))
#>   colour
#> 1   blue
#> 2   blue
#> 3   blue
#> 4  green
#> 5  green
#> 6  Other
#> 7  Other

Answer 2

with tidyverse functions, you can try something like:

df %>%
  group_by(colour) %>%
  mutate(cnt = n()) %>%
  mutate(grp = if_else(cnt >= 2, as.character(colour), as.character("Other"))) %>%
  select(-cnt)

which gives (here, the threshold value being >= 2)

  colour grp  
  <fct>  <chr>
1 blue   blue 
2 blue   blue 
3 blue   blue 
4 green  green
5 green  green
6 orange Other
7 grey   Other