I am using rvest
. And I would like to convert the result to a data frame:
> links <- pgsession %>% jump_to(urls[2]) %>% read_html() %>% html_nodes("a")
> links
{xml_nodeset (114)}
[1] <a href="/Mitglieder/Detail/1213412">Date</a>
[2] <a href="/Account/ChangePassword">Kennwort ändern</a>
[3] <a href="/Account/BenutzernamenAendern/124312234">Benutzernamen ändern</a>
[4] <a href="/Account/LogOff">Abmelden</a>
...
I used the following method:
library(plyr)
ldply(xmlToList(links), data.frame)
Error in UseMethod("xmlSApply") :
no applicable method for 'xmlSApply' applied to an object of class "xml_nodeset"
df1 <- data.frame(character(13000))
df1 <- rbind(df1, data.frame(links ))# append to data.frame
However, I get an error:
Error in UseMethod("xmlSApply") :
no applicable method for 'xmlSApply' applied to an object of class "xml_nodeset"
Any suggestion what I am doing wrong?
I appreciate your replies!
This will get you all the attributes from the links into a tbl_df
. bind_rows
gets you "fill" for free:
library(rvest)
library(dplyr)
pg <- read_html("https://en.wikipedia.org/wiki/Main_Page")
links <- html_nodes(pg, "a")
bind_rows(lapply(xml_attrs(links), function(x) data.frame(as.list(x), stringsAsFactors=FALSE)))
## Source: local data frame [310 x 10]
##
## id href title class dir accesskey rel lang hreflang style
## (chr) (chr) (chr) (chr) (chr) (chr) (chr) (chr) (chr) (chr)
## 1 top NA NA NA NA NA NA NA NA NA
## 2 NA #mw-head NA NA NA NA NA NA NA NA
## 3 NA #p-search NA NA NA NA NA NA NA NA
## 4 NA /wiki/Wikipedia Wikipedia NA NA NA NA NA NA NA
## 5 NA /wiki/Free_content Free content NA NA NA NA NA NA NA
## 6 NA /wiki/Encyclopedia Encyclopedia NA NA NA NA NA NA NA
## 7 NA /wiki/Wikipedia:Introduction Wikipedia:Introduction NA NA NA NA NA NA NA
## 8 NA /wiki/Special:Statistics Special:Statistics NA NA NA NA NA NA NA
## 9 NA /wiki/English_language English language NA NA NA NA NA NA NA
## 10 NA /wiki/Portal:Arts Portal:Arts NA NA NA NA NA NA NA
## .. ... ... ... ... ... ... ... ... ... ...
Alternately, you could use purrr
:
library(rvest)
library(purrr)
pg <- read_html("https://en.wikipedia.org/wiki/Main_Page")
html_nodes(pg, "a") %>%
map(xml_attrs) %>%
map_df(~as.list(.))
## # A tibble: 342 × 10
## id href title class dir accesskey rel hreflang lang style
## <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr>
## 1 top <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA>
## 2 <NA> #mw-head <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA>
## 3 <NA> #p-search <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA>
## 4 <NA> /wiki/Wikipedia Wikipedia <NA> <NA> <NA> <NA> <NA> <NA> <NA>
## 5 <NA> /wiki/Free_content Free content <NA> <NA> <NA> <NA> <NA> <NA> <NA>
## 6 <NA> /wiki/Encyclopedia Encyclopedia <NA> <NA> <NA> <NA> <NA> <NA> <NA>
## 7 <NA> /wiki/Wikipedia:Introduction Wikipedia:Introduction <NA> <NA> <NA> <NA> <NA> <NA> <NA>
## 8 <NA> /wiki/Special:Statistics Special:Statistics <NA> <NA> <NA> <NA> <NA> <NA> <NA>
## 9 <NA> /wiki/English_language English language <NA> <NA> <NA> <NA> <NA> <NA> <NA>
## 10 <NA> /wiki/Portal:Arts Portal:Arts <NA> <NA> <NA> <NA> <NA> <NA> <NA>
## # ... with 332 more rows
which I think is more functionally idiomatic and an overall cleaner approach.
try this out, it worked for me: The function was written by https://rdrr.io/github/lawine90/datagokR/src/R/xml_to_dataframe.R
Simply replace the link and it might work out :)
library(xml2)
xml_to_dataframe <- function(nodeset){
if(class(nodeset) != 'xml_nodeset'){
stop('Input should be "xml_nodeset" class')
}
lst <- lapply(nodeset, function(x){
tmp <- xml2::xml_text(xml2::xml_children(x))
names(tmp) <- xml2::xml_name(xml2::xml_children(x))
return(as.list(tmp))
})
result <- do.call(plyr::rbind.fill, lapply(lst, function(x)
as.data.frame(x, stringsAsFactors = F)))
return(dplyr::as.tbl(result))
}
#list of files to process
sitemap_raw <-"your-link"
sitemap <-read_xml(sitemap_raw)
sitemap_nodeset <- xml_children(sitemap)
sitemap_df <- xml_to_dataframe(sitemap_nodeset)
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With