ERROR: Test failed: 400 (InvalidToken): The provided token is malformed or otherwise invalid

Question

s3cmd was installed from epel-testing repo by running:

yum --enablerepo epel-testing install s3cmd

Then I invoked the configuration tool with s3cmd --configure but I got this error:

Test access with supplied credentials? [Y/n] 
Please wait, attempting to list all buckets...
ERROR: Test failed: 400 (InvalidToken): The provided token is malformed or otherwise invalid.

Invoked as: /usr/bin/s3cmd --configureProblem: AttributeError: 'S3Error' object has no attribute 'find'
S3cmd:   1.5.0-beta1
python:   2.6.8 (unknown, Mar 14 2013, 09:31:22) 
[GCC 4.6.2 20111027 (Red Hat 4.6.2-2)]

Traceback (most recent call last):
  File "/usr/bin/s3cmd", line 2323, in <module>
    main()
  File "/usr/bin/s3cmd", line 2221, in main
    run_configure(options.config, args)
  File "/usr/bin/s3cmd", line 1704, in run_configure
    if e.find('403') != -1:
AttributeError: 'S3Error' object has no attribute 'find'

I'm sure the keys are correct.

Do you have any idea about this?

---

UPDATE Fri Mar 21 22:44:42 ICT 2014

Found some clues when running in debug mode.

With the same credentials, on the worked system:

DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Fri, 21 Mar 2014 07:07:18 +0000\n/'

On the failed system:

DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Fri, 21 Mar 2014 07:40:56 +0000\nx-amz-security-token:AQoDYXdzENb...\n/'

This security token was taken from the metadata:

# wget -O - -q 'http://169.254.169.254/latest/meta-data/iam/security-credentials/myrole'
{
  "Code" : "Success",
  "LastUpdated" : "2014-03-21T12:45:27Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "foo",
  "SecretAccessKey" : "bar",
  "Token" : "AQoDYXdzENb....",
  "Expiration" : "2014-03-21T19:18:02Z"
}

So the my question should be changed to: why does sometimes s3cmd add x-amz-security-token to the header when running --configure (I am not using --add-header)?

Answer 1

After installing s3cmd, setting up an IAM and attaching an "Amazon S3 Full Access" role - I too encountered the "The provided token is malformed or otherwise invalid" error. So next, I created an IAM and attached a policy with admin credentials (everything).

error appeared again using this new IAM.

From a github.com thread, @mdomsch mentioned some possible issues using EC2 with an embedded IAM roles - which is what my EC2 that was using the s3cmd had.

So next, I tried running.

s3cmd --access_key=xxxx --secret_key=xxxxxxxxxxxxx ls

using the IAM keys with the S3 policy and it worked.

So s3cmd takes as a default the credentials of embedded roles in the EC2. I am sure that if you spin up another EC2 with a role that has access to S3 - you will not get this "The provided token is malformed or otherwise invalid" error.

However - please DO NOT stick -access_key=xxxx --secret_key=xxxxxxxxxxxxx parameters into any script using s3cmd. Where ever possible - try to embed roles into EC2s when you fire them up (good security practice)

Anyway - to test - I fired up another EC2 w/out any embedded IAM roles, installed s3cmd, configured - and everything worked as expected using my IAM user that had the S3 policy.

Answer 2

This held me up for an hour or two as well. The way I fixed this on an EC2 instances is by setting up my Security Credentials in the environment before configuring s3cmd.

The easiest way was to add the AWS Key and Secret as environment variables:

export AWS_ACCESS_KEY_ID=EXAMPLE_KEY
export AWS_SECRET_ACCESS_KEY=EXAMPLE_SECRET

You can also set up an aws_config_file in ~/.aws.

After your environment is set up, run s3cmd --configure and you should be set to go.

Answer 3

for those who have the same problem this is my solution: create/edit the file /home/vagrant/.s3cfg and add the your secret and key between quotes e.g:

access_key="BLAHBBLAHBBLAHBBLAHB" secret_key="2T0wyvmhfGw42T0wyvmhfGw4+-"

Answer 4

Note, this problem also occurs on Windows 10 using:

aws s3 ls s3://mybucketname/

I too have an EC2 instance assigned an IAM role with full admin access. All I did was rm the .aws/config and .aws/credentials file after creating a user with full admin privileges the type in again:

aws configure

The good old remove and replace got my back.