Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

data.table's fread() giving unwanted download message

Tags:

r

data.table

I'm not too familiar with data.table's fread function, but it makes quick work of reading my data, so now I'm intrigued. At URL "http://www.retrosheet.org/CurrentNames.csv", there is a simple csv file. The following two calls work fine.

readLines("http://www.retrosheet.org/CurrentNames.csv", n = 2)
# [1] "ANA,LAA,AL,,Los Angeles,Angels,,4/11/1961,9/1/1965,Los Angeles,CA"
# [2] "ANA,CAL,AL,,California,Angels,,9/2/1965,9/29/1968,Anaheim,CA"
rcsv <- read.csv("http://www.retrosheet.org/CurrentNames.csv", header = FALSE)

But fread is delivering a download message, and I can't seem to turn it off with

showProgress = FALSE

I could use suppressMessages(), but I don't really want to.

library(data.table)
dtf <- fread("http://www.retrosheet.org/CurrentNames.csv", 
             header = FALSE, showProgress = FALSE)
# trying URL 'http://www.retrosheet.org/CurrentNames.csv'
# Content type 'text/plain' length 7729 bytes
# opened URL
# ==================================================
# downloaded 7729 bytes

Can anyone explain this, and can I turn it off in the fread arguments?

It looks like a call to download.file has occurred somewhere. Why wouldn't fread just read the URL the same way as read.csv?

like image 700
Rich Scriven Avatar asked Jul 24 '14 02:07

Rich Scriven


1 Answers

Update Oct 2014. Now in v1.9.5 :

fread now passes showProgress=FALSE through to download.file() as quiet=!showProgress. Thanks to a pull request from Karl Broman and Richard Scriven for filing the issue, #741.


Previous answer ...

It does download the file, here is the part of the code that does it.

else if (substring(input, 1, 7) %chin% c("http://", "https:/", 
    "file://")) {
    tt = tempfile()
    on.exit(unlink(tt), add = TRUE)
    download.file(input, tt)
    input = tt
}

My guess this is because fread makes more than one pass at the file, first to get the structure, then to actually read the whole thing in. Saves downloading multiple times.

like image 107
JeremyS Avatar answered Oct 17 '22 16:10

JeremyS