Read gzipped csv directly from a url in R

Question

I'm looking to download a gzipped csv and load it as an R object without saving it first to disk. I can do this with zipped files but can't seem to get it to work with gzfile or gzcon.

Example:

grabRemote <- function(url) {
    temp <- tempfile()
    download.file(url, temp)
    aap.file <- read.csv(gzfile(temp), as.is = TRUE)
    unlink(temp)
    return(aap.file)
}
grabRemote("http://dumps.wikimedia.org/other/articlefeedback/aa_combined-20110321.csv.gz")

That downloads a (small) gz compressed file containing Wikipedia article feedback data (not important, but just to indicate it isn't giant or nefarious).

The code I have works fine but I feel like I'm missing something very obvious by resorting to creating and destroying a temporary file.

Answer 1

I am almost certain I answered this question once before. The upshot is that Connections API of R (file(), url(), pipe(), ...) can do decompression on the fly, I do not think you can do it for remote http objects.

So do the very two-step you have described: use download.file() with a tempfile() result as second argument to fetch the compressed file, and then read from it. As tempfile() object, it will get cleaned up automatically at the end of your R session so the one minor fix I can suggest is to skip the unlink() (but then I like explicit cleanups, so you may as well keep it).

Edit: Got it:

con <- gzcon(url(paste("http://dumps.wikimedia.org/other/articlefeedback/",
                       "aa_combined-20110321.csv.gz", sep="")))
txt <- readLines(con)
dat <- read.csv(textConnection(txt))

dim(dat)
# [1] 1490   19

summary(dat[,1:3])
# aa_page_id       page_namespace                 page_title  
# Min.   :     324   Min.   :0      United_States        :  79  
# 1st Qu.:   88568   1st Qu.:0      2011_NBA_Playoffs    :  52  
# Median : 2445733   Median :0      IPad_2               :  43  
# Mean   : 8279600   Mean   :0      IPod_Touch           :  38  
# 3rd Qu.:16179920   3rd Qu.:0      True_Grit_(2010_film):  38  
# Max.   :31230028   Max.   :0      IPhone_4             :  26  
# (Other)              :1214  

The key was the hint the gzcon help that it can put decompression around an existing stream. We then need the slight detour of readLines and reading via textConnection from that as read.csv wants to go back and forth in the data (to validate column width, I presume).

Answer 2

Using data.table::fread :

x <- data.table::fread("http://dumps.wikimedia.org/other/articlefeedback/aa_combined-20110321.csv.gz")

dim(x)
[1] 1490   19

x[, 1:2]
#       aa_page_id page_namespace
#    1:   26224556              0
#    2:      31653              0
#    3:   26224556              0
#    4:   26224556              0
#    5:    1058990              0
#   ---                          
# 1486:     619464              0
# 1487:   19283361              0
# 1488:   19006979              0
# 1489:    5078775              0
# 1490:   30209619              0

Answer 3

This function generalizes Dirk's answer:

R <- function(file_url) {
  con <- gzcon(url(file_url))
  txt <- readLines(con)
  return(read.csv(textConnection(txt)))
}