How do you use an HTTP/HTTPS proxy with boto3?

Question

On the old boto library is was simple enough to use the proxy, proxy_port, proxy_user and proxy_pass parameters when you open a connection. However, I could not find any equivalent way of programmatically define the proxy parameters on boto3. :(

Answer 1

As of at least version 1.5.79, botocore accepts a proxies argument in the botocore config.

e.g.

import boto3
from botocore.config import Config

boto3.resource('s3', config=Config(proxies={'https': 'foo.bar:3128'}))

boto3 resource https://boto3.readthedocs.io/en/latest/reference/core/session.html#boto3.session.Session.resource

botocore config https://botocore.readthedocs.io/en/stable/reference/config.html#botocore.config.Config

Answer 2

If you user proxy server does not have a password try the following:

import os
os.environ["HTTP_PROXY"] = "http://proxy.com:port"
os.environ["HTTPS_PROXY"] = "https://proxy.com:port"

if you user proxy server has a password try the following:

import os
os.environ["HTTP_PROXY"] = "http://user:[email protected]:port"
os.environ["HTTPS_PROXY"] = "https://user:[email protected]:port"

Answer 3

Apart from altering the environment variable, I'll present what I found in the code.

Since boto3 uses botocore, I had a look through the source code:

https://github.com/boto/botocore/blob/66008c874ebfa9ee7530d944d274480347ac3432/botocore/endpoint.py#L265

From this link, we end up at:

    def _get_proxies(self, url):
        # We could also support getting proxies from a config file,
        # but for now proxy support is taken from the environment.
        return get_environ_proxies(url)

...which is called by proxies = self._get_proxies(final_endpoint_url) in the EndpointCreator class.

Long story short, if you're using python2 it will use the getproxies method from urllib2 and if you're using python3, it will use urllib3.

get_environ_proxies is expecting a dict containing {'http:' 'url'} (and I'm guessing https too).

You could always patch the code, but that is poor practice.

Answer 4

This is one of the rare occasions when I would recommend monkey-patching, at least until the Boto developers allow connection-specific proxy settings:

import botocore.endpoint
def _get_proxies(self, url):
    return {'http': 'http://someproxy:1234/', 'https': 'https://someproxy:1234/'}
botocore.endpoint.EndpointCreator._get_proxies = _get_proxies
import boto3