Setting a User Agent in scrape-it

Question

I'm using scrape-it in my node.js scraping tool (for identifying proper keyword usage) but being identified as a bot by some websites and not getting any content. Is there a way to configure a known user agent header for the GET request to bypass the block?

Answer 1

You can set the headers, including User-agent, by passing an options object to scrape-it:

scrapeIt({
    url: "http://example.com"
  , headers: { "User-agent": "known-user-agent-of-choice" }
},
{
  // some scrapeHTML options ...
})
.then(
 // some code ...
);