Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Scraping javascript with R

I want to download tables from metal-archives.com, exactly from http://www.metal-archives.com/artist/rip, but there is one big problem. This tables are generated by javascript. In fact I don't know what to do in this case.

Is there a possibility to parse this site with R and XML package?

like image 500
Maciej Avatar asked Aug 08 '26 09:08

Maciej


2 Answers

Here's all information in JSON format

http://www.metal-archives.com/artist/ajax-rip

like image 77
bumbu Avatar answered Aug 09 '26 21:08

bumbu


Thanks to user bubmu I achieve what I wanted. Below is code, which solves my problem.

a<-1:8
b<-200*a
x<-paste("http://www.metal-archives.com/artist/ajax-rip?iDisplayStart=",b,"&sEcho=",a,sep="")
x<-c(x,"http://www.metal-archives.com/artist/ajax-rip?iDisplayStart=1700&sEcho=9")

JSONparse<-function(x){
  library(XML)
  doc<-htmlParse(x)
  str<-xpathApply(doc,'//p',xmlValue)[[1]][1]
  x1<-strsplit(str,'\\[')
  x1<-x1[[1]][-1]
  x1<-x1[-1]

  x2<-strsplit(x1,'\\",')
  x3<-lapply(x2, function(y) {
    y<-gsub('\\t','',y)
    y<-gsub('\\n','',y)
    y<-gsub('\\r','',y)
    y<-gsub('\\\"','',y)
    y<-gsub('\\]}','',y)
    y<-gsub('\\],','',y)
    y<-as.data.frame(t(y))
    y})

  allinall<-do.call('rbind',x3)
  colnames(allinall)<-c("Artist","Country","Band","When","Why")
  allinall
}

metallum<-lapply(x,JSONparse)
metallum<-do.call('rbind',metallum)

But it works only for this site. Of course better is RJSONIO or rjson package.

like image 44
Maciej Avatar answered Aug 09 '26 23:08

Maciej



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!