Is amazon lambda suitable for web scraping?

Question

If I create a function to get webpages. Will it execute it on different IP per execution so that my scraping requests dont get blocked?

Answer 1

Yes, lambda by default executes with random IPs. You can trigger it using things like event bridge so you can have a schedule to execute the script every hour or similar. Others can possibly recommend using API Gateway, however, it is highly insecure to expose API endpoints available for anyone to trigger. So you have to write additional logic to protect it either by hard coded headers or say oauth.