NoCredentialsError : Unable to locate credentials - python module boto3

Question

I am running django in a python virtual environment(virtualenv). The django website is served by apache2 from an amazon ec2 instance(ubuntu 16.04). I use boto3 module to write to amazon s3.

I installed awscli and ran aws configure and set up my aws access keys correctly. ( I know I configured it correctly, because $ aws s3 ls returns the correct lists of my s3 buckets.)

However, when I try to write some objects to s3 from django application, it fails producing the error as described in the title.

I recently moved to a new instance and started using python virtual environments. Before that, it used to work fine. I have read the questions on SO and the docs from aws. Below is the stack trace.

Environment:


Request Method: POST
Request URL: http://*******/product/4

Django Version: 1.10.6
Python Version: 3.5.2
Installed Applications:
('django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'abc.apps.abcdirectConfig')
Installed Middleware:
('django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware')



Traceback:

File "/home/ubuntu/.virtualenv/lib/python3.5/site-packages/django/core/handlers/exception.py" in inner
  42.             response = get_response(request)

File "/home/ubuntu/.virtualenv/lib/python3.5/site-packages/django/core/handlers/base.py" in _legacy_get_response
  249.             response = self._get_response(request)

File "/home/ubuntu/.virtualenv/lib/python3.5/site-packages/django/core/handlers/base.py" in _get_response
  187.                 response = self.process_exception_by_middleware(e, request)

File "/home/ubuntu/.virtualenv/lib/python3.5/site-packages/django/core/handlers/base.py" in _get_response
  185.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/home/ubuntu/.virtualenv/lib/python3.5/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  23.                 return view_func(request, *args, **kwargs)

File "/home/ubuntu/abcdirect/abcdirect/views.py" in view_product
  385.             s3.Bucket('abccms').put_object(Key=s3_file_name, Body=s3_file_data)

File "/home/ubuntu/.virtualenv/lib/python3.5/site-packages/boto3/resources/factory.py" in do_action
  520.                 response = action(self, *args, **kwargs)

File "/home/ubuntu/.virtualenv/lib/python3.5/site-packages/boto3/resources/action.py" in __call__
  83.         response = getattr(parent.meta.client, operation_name)(**params)

File "/home/ubuntu/.virtualenv/lib/python3.5/site-packages/botocore/client.py" in _api_call
  253.             return self._make_api_call(operation_name, kwargs)

File "/home/ubuntu/.virtualenv/lib/python3.5/site-packages/botocore/client.py" in _make_api_call
  530.                 operation_model, request_dict)

File "/home/ubuntu/.virtualenv/lib/python3.5/site-packages/botocore/endpoint.py" in make_request
  141.         return self._send_request(request_dict, operation_model)

File "/home/ubuntu/.virtualenv/lib/python3.5/site-packages/botocore/endpoint.py" in _send_request
  166.         request = self.create_request(request_dict, operation_model)

File "/home/ubuntu/.virtualenv/lib/python3.5/site-packages/botocore/endpoint.py" in create_request
  150.                                      operation_name=operation_model.name)

File "/home/ubuntu/.virtualenv/lib/python3.5/site-packages/botocore/hooks.py" in emit
  227.         return self._emit(event_name, kwargs)

File "/home/ubuntu/.virtualenv/lib/python3.5/site-packages/botocore/hooks.py" in _emit
  210.             response = handler(**kwargs)

File "/home/ubuntu/.virtualenv/lib/python3.5/site-packages/botocore/signers.py" in handler
  90.         return self.sign(operation_name, request)

File "/home/ubuntu/.virtualenv/lib/python3.5/site-packages/botocore/signers.py" in sign
  147.             auth.add_auth(request)

File "/home/ubuntu/.virtualenv/lib/python3.5/site-packages/botocore/auth.py" in add_auth
  679.             raise NoCredentialsError

Exception Type: NoCredentialsError at /product/4
Exception Value: Unable to locate credentials

Answer 1

I figured out the reason I was getting the error. I am posting the answer just in case someone else encounters this issue.

tl;dr : aws config files did not live in apache's home directory

The django app was running under the user www-data(apache2). When I configured my credentials using aws configure, my settings were stored in .aws/config file.

Now this is where the problem was.

The .aws/configure file was stored in my home directory and not in the home directory of www-data(apache2),which is /var/www by default. So when the django app called boto3 module, the module was looking for the config file in /var/www/.aws/config but my files were in /home/ubuntu/.aws/config.

Simply copying the relevant files to /var/www/ fixed the problem for me.