Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to get all the URLs in a web site using JavaScript?

Tags:

javascript

url

Any one knows a way to get all the URLs in a website using JavaScript?

I only need the links starting with the same domain name.no need to consider other links.

like image 277
netha Avatar asked Sep 29 '10 17:09

netha


1 Answers

Well this will get all the same-host links on the page:

var urls = [];
for(var i = document.links.length; i --> 0;)
    if(document.links[i].hostname === location.hostname)
        urls.push(document.links[i].href);

If by site you mean you want to recursively get the links inside linked pages, that's a bit trickier. You'd have to download each link into a new document (for example in an <iframe>), and the onload check the iframe's own document for more links to add to the list to fetch. You'd need to keep a lookup of what URLs you'd already spidered to avoid fetching the same document twice. It probably wouldn't be very fast.

like image 128
bobince Avatar answered Sep 21 '22 17:09

bobince