Scrapy Torproject

Question

On Ubuntu,

[guy@guy-laptop tmp]$ sudo /etc/init.d/tor status
tor is running
[guy@guy-laptop tmp]$ sudo /etc/init.d/polipo start
Starting polipo: /usr/bin/polipo already running -- doing nothing
polipo.

also:

$ python -c 'import urllib; print urllib.getproxies()'
{'ftp': 'ftp://127.0.0.1:8118/', 'all': 'socks://127.0.0.1:8118/',
'http': 'http://127.0.0.1:8118/', 'https': 'https://127.0.0.1:8118/',
'no': 'localhost,127.0.0.0/8,*.local'}

When running scrapy I get:

ERROR: Error downloading https://registration.example.com/login.fcc: [Failure instance: Traceback (failure with no frames): : [('SSL routines', 'SSL23_READ', 'ssl handshake failure')] ]

While on the same time FireFox managed to get the page correctly through proxy

Any help will be appreciated, Thanks, Guy

Answer 1

Have you tried running the script through proxychains? To install on ubuntu:

:~$ sudo apt-get install proxychains

then configure the /etc/proxychains.conf file to work with TOR (socks4/5).

# defaults set to "tor"

socks4 127.0.0.1 9050

Then you can run anything though TOR

:~$ proxychains scriptwhatever.py target

Once you know TOR is working correctly, I recommend quiet mode in the proxychains.conf file.

# Quiet mode (no output from library).
quiet_mode