S3Cmd doesn't work with S3 Ninja

Question

I'm trying to use my local s3ninja with s3cmd. Every command like: s3cmd ls s3://test throws the same exceptions.

The s3cfg seems to be ok and the called endpoints are correct.

Was anyone able to use s3ninja with s3cmd?

PS: I know S3 isn't costly and there are many better ways to test against S3. I need S3 Ninja for other reasons.

Here is the debug information:

DEBUG: ConfigParser: Reading file '/Users/daniel/.s3cfg'
DEBUG: ConfigParser: access_key->AK...17_chars...E
DEBUG: ConfigParser: access_token->
DEBUG: ConfigParser: add_encoding_exts->
DEBUG: ConfigParser: add_headers->
DEBUG: ConfigParser: bucket_location->US
DEBUG: ConfigParser: ca_certs_file->
DEBUG: ConfigParser: cache_file->
DEBUG: ConfigParser: check_ssl_certificate->True
DEBUG: ConfigParser: check_ssl_hostname->True
DEBUG: ConfigParser: cloudfront_host->cloudfront.amazonaws.com
DEBUG: ConfigParser: default_mime_type->binary/octet-stream
DEBUG: ConfigParser: delay_updates->False
DEBUG: ConfigParser: delete_after->False
DEBUG: ConfigParser: delete_after_fetch->False
DEBUG: ConfigParser: delete_removed->False
DEBUG: ConfigParser: dry_run->False
DEBUG: ConfigParser: enable_multipart->True
DEBUG: ConfigParser: encoding->UTF-8
DEBUG: ConfigParser: encrypt->False
DEBUG: ConfigParser: expiry_date->
DEBUG: ConfigParser: expiry_days->
DEBUG: ConfigParser: expiry_prefix->
DEBUG: ConfigParser: follow_symlinks->False
DEBUG: ConfigParser: force->False
DEBUG: ConfigParser: get_continue->False
DEBUG: ConfigParser: gpg_command->None
DEBUG: ConfigParser: gpg_decrypt->%(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_encrypt->%(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_passphrase->...-3_chars...
DEBUG: ConfigParser: guess_mime_type->True
DEBUG: ConfigParser: host_base->s3.mylocalaws.com:9444/s3
DEBUG: ConfigParser: host_bucket->s3.mylocalaws.com:9444/s3/%(bucket)s
DEBUG: ConfigParser: human_readable_sizes->False
DEBUG: ConfigParser: invalidate_default_index_on_cf->False
DEBUG: ConfigParser: invalidate_default_index_root_on_cf->True
DEBUG: ConfigParser: invalidate_on_cf->False
DEBUG: ConfigParser: kms_key->
DEBUG: ConfigParser: limitrate->0
DEBUG: ConfigParser: list_md5->False
DEBUG: ConfigParser: log_target_prefix->
DEBUG: ConfigParser: long_listing->False
DEBUG: ConfigParser: max_delete->-1
DEBUG: ConfigParser: mime_type->
DEBUG: ConfigParser: multipart_chunk_size_mb->15
DEBUG: ConfigParser: preserve_attrs->True
DEBUG: ConfigParser: progress_meter->True
DEBUG: ConfigParser: proxy_host->
DEBUG: ConfigParser: proxy_port->0
DEBUG: ConfigParser: put_continue->False
DEBUG: ConfigParser: recursive->False
DEBUG: ConfigParser: recv_chunk->65536
DEBUG: ConfigParser: reduced_redundancy->False
DEBUG: ConfigParser: requester_pays->False
DEBUG: ConfigParser: restore_days->1
DEBUG: ConfigParser: secret_key->wJ...37_chars...Y
DEBUG: ConfigParser: send_chunk->65536
DEBUG: ConfigParser: server_side_encryption->False
DEBUG: ConfigParser: signature_v2->False
DEBUG: ConfigParser: simpledb_host->sdb.amazonaws.com
DEBUG: ConfigParser: skip_existing->False
DEBUG: ConfigParser: socket_timeout->300
DEBUG: ConfigParser: stop_on_error->False
DEBUG: ConfigParser: storage_class->
DEBUG: ConfigParser: urlencoding_mode->normal
DEBUG: ConfigParser: use_https->False
DEBUG: ConfigParser: use_mime_magic->True
DEBUG: ConfigParser: verbosity->WARNING
DEBUG: ConfigParser: website_endpoint->http://%(bucket)s.s3-website-%(location)s.amazonaws.com/
DEBUG: ConfigParser: website_error->
DEBUG: ConfigParser: website_index->index.html
DEBUG: Updating Config.Config cache_file -> 
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Updating Config.Config verbosity -> 10
DEBUG: Unicodising 'ls' using UTF-8
DEBUG: Unicodising 's3://test' using UTF-8
DEBUG: Command: ls
DEBUG: Bucket 's3://test':
DEBUG: CreateRequest: resource[uri]=/
DEBUG: Using signature v4
DEBUG: get_hostname(test): s3.mylocalaws.com:9444/s3/test
DEBUG: canonical_headers = host:s3.mylocalaws.com:9444/s3/test
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20160718T104632Z

DEBUG: Canonical Request:
GET
/
delimiter=%2F
host:s3.mylocalaws.com:9444/s3/test
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20160718T104632Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
----------------------
DEBUG: signature-v4 headers: {'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': 'AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20160718/US/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=f58c63587c00d0c01122db06cf0b8a7376a2484b0572db3c562edddbd6d2fdb1', 'x-amz-date': '20160718T104632Z'}
DEBUG: Processing request, please wait...
DEBUG: get_hostname(test): s3.mylocalaws.com:9444/s3/test
DEBUG: ConnMan.get(): creating new connection: http://s3.mylocalaws.com:9444/s3/test

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
  Please try reproducing the error using
  the latest s3cmd code from the git master
  branch found at:
    https://github.com/s3tools/s3cmd
  and have a look at the known issues list:
    https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions
  If the error persists, please report the
  following lines (removing any private
  info as necessary) to:
   s3tools-bugs@lists.sourceforge.net

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

DEBUG: Unicodising '/usr/local/bin/s3cmd' using UTF-8
DEBUG: Unicodising 'ls' using UTF-8
DEBUG: Unicodising 's3://test' using UTF-8
DEBUG: Unicodising '--debug' using UTF-8
Invoked as: /usr/local/bin/s3cmd ls s3://test --debug
Problem: UnboundLocalError: local variable 'conn' referenced before assignment
S3cmd:   1.6.0
python:   2.7.11 (default, Jan 22 2016, 08:29:18) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)]
environment LANG=de_DE.UTF-8

Traceback (most recent call last):
  File "/usr/local/bin/s3cmd", line 2805, in <module>
    rc = main()
  File "/usr/local/bin/s3cmd", line 2713, in main
    rc = cmd_func(args)
  File "/usr/local/bin/s3cmd", line 120, in cmd_ls
    subcmd_bucket_list(s3, uri)
  File "/usr/local/bin/s3cmd", line 153, in subcmd_bucket_list
    response = s3.bucket_list(bucket, prefix = prefix)
  File "/usr/local/lib/python2.7/site-packages/S3/S3.py", line 293, in bucket_list
    for dirs, objects in self.bucket_list_streaming(bucket, prefix, recursive, uri_params):
  File "/usr/local/lib/python2.7/site-packages/S3/S3.py", line 320, in bucket_list_streaming
    response = self.bucket_list_noparse(bucket, prefix, recursive, uri_params)
  File "/usr/local/lib/python2.7/site-packages/S3/S3.py", line 339, in bucket_list_noparse
    response = self.send_request(request)
  File "/usr/local/lib/python2.7/site-packages/S3/S3.py", line 1084, in send_request
    conn.counter = ConnMan.conn_max_counter
UnboundLocalError: local variable 'conn' referenced before assignment

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
  Please try reproducing the error using
  the latest s3cmd code from the git master
  branch found at:
    https://github.com/s3tools/s3cmd
  and have a look at the known issues list:
    https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions
  If the error persists, please report the
  above lines (removing any private
  info as necessary) to:
   s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

kwarunek · Accepted Answer

s3cmd at the moment forces api to be at /. On the other hand s3ninja does not allow to set it to '/'. I've proposed feature (PR 781) to s3cmd to deal with - you could patch s3cmd by yourself.

Another solution would be to set some proxy e.g. nginx / > /s3

Daniel R. · Answer

This is my nginx proxy configuration that works perfectly with s3cmd and S3 Ninja.

server {
      listen 80;
      server_name s3.eu-central-1.amazonaws.com;

      location / {
          proxy_pass         http://127.0.0.1:9444/s3/;
      }
  }

With the following host configuration:

host_base = s3.eu-central-1.amazonaws.com
host_bucket = s3.eu-central-1.amazonaws.com

And you need to add an entry to /etc/hosts:

127.0.0.1  s3.eu-central-1.amazonaws.com

This also works with a reverse proxy and https.

S3Cmd doesn't work with S3 Ninja

Tags:

python

amazon-web-services

amazon-s3

s3cmd

Daniel R.

2 Answers

kwarunek

Daniel R.

Recent Activity

Donate For Us

S3Cmd doesn't work with S3 Ninja

Tags:

python

amazon-web-services

amazon-s3

s3cmd

Daniel R.

2 Answers

kwarunek

Daniel R.

Related questions

Recent Activity

Donate For Us