Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is it possible to do a full join in dplyr and keep all the columns used in the join?

Tags:

r

dplyr

sparklyr

I have two tables that I want to do a full join using dplyr, but I don't want it to drop any of the columns. Per the documentation and my own experience it is only keeping the join column for the left hand side. This is a problem when you have a row with a record for the right hand side since the join value is gone.

For example, suppose I have the two tables a and b,

customerId | revenue               customerId | state
-----------|---------              -----------|-------
    1      | 2000                       1     |  CA
    2      | 3000                       3     |  GA
    4      | 4000                       4     |  NY

doing something like full_join(a, b, by="customerId") will produce

customerId | revenue | state
-----------|---------|-------
    1      |   2000  |  CA
    2      |   3000  | <NA>
   <NA>    |   <NA>  |  GA
    4      |   4000  |  NY

so there is no way to tell which customer that third row is from. The ideal output would be

customerId.a | customerId.b | revenue | state
-------------|--------------|---------|-------
      1      |      1       |   2000  |  CA
      2      |     <NA>     |   3000  | <NA>
    <NA>     |      3       |   <NA>  |  GA
      4      |      4       |   4000  |  NY

note that this is just a toy example. I'm actually using sparklyr so this is all being run in Spark. Thus, merge won't work here for me. Is there a way to do what I'm looking for in dplyr?

EDIT: As someone pointed out this actually is working as desired in dplyr itself locally. However, I do see this problem using sparklyr (which uses dplyr). Here is the code to see that:

library(sparklyr)
sc <- spark_connect("local[4]")
d1 <- data_frame(customerId = c("1","2","4"), revenue=c(2000,3000,4000))
d2 <- data_frame(customerId = c("1","3","4"), state=c("CA", "GA", "NY"))
d1_tbl <- copy_to(sc, d1)
d2_tbl <- copy_to(sc, d2)
full_join(d1_tbl, d2_tbl, by=c("customerId"))
like image 912
Dave Kincaid Avatar asked Dec 04 '22 22:12

Dave Kincaid


1 Answers

You can create separate identical customerId for both data frames before join:

full_join(
    mutate(a, customerId.a = customerId), 
    mutate(b, customerId.b = customerId), 
    by="customerId"
) %>% select(-customerId)

#  revenue customerId.a state customerId.b
#1    2000            1    CA            1
#2    3000            2  <NA>           NA
#3    4000            4    NY            4
#4      NA           NA    GA            3
like image 173
Psidom Avatar answered Dec 11 '22 15:12

Psidom