I have two tables that I want to do a full join using dplyr, but I don't want it to drop any of the columns. Per the documentation and my own experience it is only keeping the join column for the left hand side. This is a problem when you have a row with a record for the right hand side since the join value is gone.
For example, suppose I have the two tables a and b,
customerId | revenue customerId | state
-----------|--------- -----------|-------
1 | 2000 1 | CA
2 | 3000 3 | GA
4 | 4000 4 | NY
doing something like full_join(a, b, by="customerId")
will produce
customerId | revenue | state
-----------|---------|-------
1 | 2000 | CA
2 | 3000 | <NA>
<NA> | <NA> | GA
4 | 4000 | NY
so there is no way to tell which customer that third row is from. The ideal output would be
customerId.a | customerId.b | revenue | state
-------------|--------------|---------|-------
1 | 1 | 2000 | CA
2 | <NA> | 3000 | <NA>
<NA> | 3 | <NA> | GA
4 | 4 | 4000 | NY
note that this is just a toy example. I'm actually using sparklyr so this is all being run in Spark. Thus, merge won't work here for me. Is there a way to do what I'm looking for in dplyr?
EDIT: As someone pointed out this actually is working as desired in dplyr itself locally. However, I do see this problem using sparklyr (which uses dplyr). Here is the code to see that:
library(sparklyr)
sc <- spark_connect("local[4]")
d1 <- data_frame(customerId = c("1","2","4"), revenue=c(2000,3000,4000))
d2 <- data_frame(customerId = c("1","3","4"), state=c("CA", "GA", "NY"))
d1_tbl <- copy_to(sc, d1)
d2_tbl <- copy_to(sc, d2)
full_join(d1_tbl, d2_tbl, by=c("customerId"))
You can create separate identical customerId
for both data frames before join:
full_join(
mutate(a, customerId.a = customerId),
mutate(b, customerId.b = customerId),
by="customerId"
) %>% select(-customerId)
# revenue customerId.a state customerId.b
#1 2000 1 CA 1
#2 3000 2 <NA> NA
#3 4000 4 NY 4
#4 NA NA GA 3
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With