ValueError: Invalid endpoint: https://s3..amazonaws.com

Question

When EMR machine is trying to run a step that includes boto3 initialisation it sometimes get the following error: ValueError: Invalid endpoint: https://s3..amazonaws.com When I'm trying to set up a new machine it can suddenly work. Attached the full error:

self.client = boto3.client("s3")
  File "/usr/local/lib/python3.6/site-packages/boto3/__init__.py", line 83, in client
    return _get_default_session().client(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/boto3/session.py", line 263, in client
    aws_session_token=aws_session_token, config=config)
  File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 861, in create_client
    client_config=config, api_version=api_version)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 76, in create_client
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 285, in _get_client_args
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/usr/local/lib/python3.6/site-packages/botocore/args.py", line 79, in get_client_args
    timeout=(new_config.connect_timeout, new_config.read_timeout))
  File "/usr/local/lib/python3.6/site-packages/botocore/endpoint.py", line 297, in create_endpoint
    raise ValueError("Invalid endpoint: %s" % endpoint_url)
ValueError: Invalid endpoint: https://s3..amazonaws.com 

Any idea why it happens?

(Versions: boto3==1.7.29, botocore==1.10.29)

Answer 1

Set the region in your ~/.aws/credentials or ~/.aws/config files. You can set the region as an environment variable as well e.g.

In bash

export AWS_REGION="eu-west-2"

or in Powershell

$Env:AWS_REGION="eu-west-2"

Answer 2

It looks like you have an invalid region. Check your ~/.aws/config