Menu
I've got a python web crawler and I want to distribute the download requests among many different proxy servers, probably running squid (though I'm open to alternatives). For example, it could work in a round-robin fashion, where request1 goes to proxy1, request2 to proxy2, and eventually looping back around. Any idea how to set this up?
To make it harder, I'd also like to be able to dynamically change the list of available proxies, bring some down, and add others.
If it matters, IP addresses are assigned dynamically.
Thanks :)
340
Rotating proxies can provide an extra level of security and anonymity as the requests you send to servers are initiated from various IP addresses, often from unrelated geo-locations. Proxies may be legal when they are used without breaching any laws.
How to use a rotating proxy? Rotating IP addresses, or rotating IP proxy, change your IP address at each request you send to the target. The easiest way to quickly implement and start using a rotating proxy is by buying a residential proxy service.
It is vital to keep in mind that when you need to use an IP address for a continuous duration, proxies with sticky sessions are preferred. In contrast, you would use a rotating proxy when you can not use the same IP address for a longer duration.
Web scrapers use proxies to hide their identity and make their traffic look like regular user traffic. Web users use proxies to protect their personal data or access websites that are blocked by their country's censorship mechanism.
I've setted up rotating proxies using HAProxy + DeleGate + Multiple Tor Instances. With Tor you don't have good control of bandwidth and latency but it's useful for web scraping. I've just published an article on the subject: Running Your Own Anonymous Rotating Proxies
172
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
