Multiple web table mining with R, RCurl

Question

First of all, thanks in advance for any responses.

I need to obtain a table by joining some smaller tables in their respective web pages. To date, I've been capable of extracting the info, but failed to do it automatically using a loop. To date, my commands are:

library(RCurl)
library(XML)
# index <- toupper(letters)
# EDIT:
index <- LETTERS

index[1] <- "0-A"
url <- paste("www.citefactor.org/journal-impact-factor-list-2014_", index, ".html", sep="", collapse=";")
urls <- strsplit(url, ";") [[1]]

Here is my loop attempt:

read.html.tab <- function(url){
 require(RCurl)
 require(XML)
 uri <- url
 tabs <- NULL
 for (i in uri){
  tabs <- getURL(uri)
  tabs <- readHTMLTable(tabs, stringsAsFactors = F)
  tab1 <- as.data.frame(tabs)
  }
 tab1
 }

If I try to use the read.html.tab function:

tab0 <- read.html.tab(urls)

I get the following error: Error in data.frame(`Search Journal Impact Factor List 2014` = list(`0-A` = "N", : arguments imply differing number of rows: 1, 1100, 447, 874, 169, 486, 201, 189, 172, 837....

However, if urls has only one element, the function works:

tabA <- read.html.tab(urls[1])
tabB <- read.html.tab(urls[2]) 
tab.if <- rbind(tabA,tabB)

ifacs <- tab.if[,27:ncol(tab.if)]
View(ifacs)

It seems I'm not understanding how loops work...

Answer 1

Obligatory Hadleyverse answer:

library(rvest)
library(dplyr)
library(magrittr)
library(pbapply)

urls <- sprintf("http://www.citefactor.org/journal-impact-factor-list-2014_%s.html", 
                c("0-A", LETTERS[-1]))

dat <- urls %>%
  pblapply(function(url) 
    html(url) %>% html_table(header=TRUE) %>% extract2(2)) %>%
  bind_rows()

glimpse(dat)

## Observations: 1547
## Variables:
## $ INDEX     (int) 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,...
## $ JOURNAL   (chr) "4OR-A Quarterly Journal of Operations Researc...
## $ ISSN      (chr) "1619-4500", "0891-0162", "0149-1423", "1550-7...
## $ 2013/2014 (chr) "0.918", "0.608", "1.832", "3.905", "1.776", "...
## $ 2012      (chr) "0.73", "0.856", "1.768", "4.386", "1.584", "0...
## $ 2011      (chr) "0.323", "0.509", "1.831", "5.086", "1.432", "...
## $ 2010      (chr) "0.69", "0.56", "1.964", "3.942", "1.211", "0....
## $ 2009      (chr) "0.75", "-", "1.448", "3.54", "1.19", "0.293",...
## $ 2008      (chr) "-", "-", "1.364", "-", "1.445", "0.352", "1.4...

rvest gives us html and html_table

I use magrittr solely for extract2, which just wraps [[ and reads better (IMO).

The pbapply package wraps the *apply functions and gives you free progress bars.

NOTE: bind_rows is in the latest dplyr, so grab that before using it.