Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How do you use an HTTP/HTTPS proxy with boto3?

Tags:

proxy

boto3

On the old boto library is was simple enough to use the proxy, proxy_port, proxy_user and proxy_pass parameters when you open a connection. However, I could not find any equivalent way of programmatically define the proxy parameters on boto3. :(

like image 858
asieira Avatar asked Nov 02 '15 14:11

asieira


People also ask

What is Boto3 resource?

Boto3 resource is a high-level object-oriented API service you can use to connect and access your AWS resource. It has actions() defined which can be used to make calls to the AWS service.


4 Answers

As of at least version 1.5.79, botocore accepts a proxies argument in the botocore config.

e.g.

import boto3
from botocore.config import Config

boto3.resource('s3', config=Config(proxies={'https': 'foo.bar:3128'}))

boto3 resource https://boto3.readthedocs.io/en/latest/reference/core/session.html#boto3.session.Session.resource

botocore config https://botocore.readthedocs.io/en/stable/reference/config.html#botocore.config.Config

like image 139
cblakkan Avatar answered Oct 08 '22 10:10

cblakkan


If you user proxy server does not have a password try the following:

import os
os.environ["HTTP_PROXY"] = "http://proxy.com:port"
os.environ["HTTPS_PROXY"] = "https://proxy.com:port"

if you user proxy server has a password try the following:

import os
os.environ["HTTP_PROXY"] = "http://user:[email protected]:port"
os.environ["HTTPS_PROXY"] = "https://user:[email protected]:port"
like image 34
user881703 Avatar answered Oct 08 '22 08:10

user881703


Apart from altering the environment variable, I'll present what I found in the code.

Since boto3 uses botocore, I had a look through the source code:

https://github.com/boto/botocore/blob/66008c874ebfa9ee7530d944d274480347ac3432/botocore/endpoint.py#L265

From this link, we end up at:

    def _get_proxies(self, url):
        # We could also support getting proxies from a config file,
        # but for now proxy support is taken from the environment.
        return get_environ_proxies(url)

...which is called by proxies = self._get_proxies(final_endpoint_url) in the EndpointCreator class.

Long story short, if you're using python2 it will use the getproxies method from urllib2 and if you're using python3, it will use urllib3.

get_environ_proxies is expecting a dict containing {'http:' 'url'} (and I'm guessing https too).

You could always patch the code, but that is poor practice.

like image 3
Mani Avatar answered Oct 08 '22 10:10

Mani


This is one of the rare occasions when I would recommend monkey-patching, at least until the Boto developers allow connection-specific proxy settings:

import botocore.endpoint
def _get_proxies(self, url):
    return {'http': 'http://someproxy:1234/', 'https': 'https://someproxy:1234/'}
botocore.endpoint.EndpointCreator._get_proxies = _get_proxies
import boto3
like image 2
someben Avatar answered Oct 08 '22 10:10

someben