How to download from AWS S3 using golang

Question

I am writing a go function to download a file from AWS S3 bucket.

func DownloadFromS3Bucket() {
    bucket := "cellery-runtime-installation"
    item := "hello-world.txt"

    file, err := os.Create(item)
    if err != nil {
        fmt.Println(err)
    }

    defer file.Close()

    // Initialize a session in us-west-2 that the SDK will use to load
    // credentials from the shared credentials file ~/.aws/credentials.
    sess, _ := session.NewSession(&aws.Config{
        Region: aws.String("us-east-1")},
    )

    downloader := s3manager.NewDownloader(sess)

    numBytes, err := downloader.Download(file,
        &s3.GetObjectInput{
            Bucket: aws.String(bucket),
            Key:    aws.String(item),
        })
    if err != nil {
        fmt.Println(err)
    }

    fmt.Println("Downloaded", file.Name(), numBytes, "bytes")
}

However, I am getting an error message asking for credentials.

NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors

The documentation does not specifically say how to set the credentials. (Access key ID,Secret access key)

Any idea?

Answer 1

There are several ways to set credentials. For more details aws/credentials.

For example, you can specify it by setting environment variables:

AWS_ACCESS_KEY = <your_access_key>
AWS_SECRET_KEY = <your_secret_key>

Then just use credentials.NewEnvCredentials() in your config instance:

sess, _ := session.NewSession(&aws.Config{
    Region:      aws.String("us-east-1"),
    Credentials: credentials.NewEnvCredentials(),
})

Answer 2

Just set AWS_ACCESS_KEY and AWS_SECRET_KEY environment variables (you do not need to call credentials.NewEnvCredentials() as part of session.NewSession - what you did is perfect):

func DownloadFromS3Bucket() {
    
    os.Setenv("AWS_ACCESS_KEY","my-key")
    os.Setenv("AWS_SECRET_KEY","my-secret")

    bucket := "cellery-runtime-installation"
    item := "hello-world.txt"

    file, err := os.Create(item)
    if err != nil {
        fmt.Println(err)
    }
    defer file.Close()

    sess, _ := session.NewSession(&aws.Config{Region: aws.String("us-east-1")})
    downloader := s3manager.NewDownloader(sess)
    numBytes, err := downloader.Download(file,
        &s3.GetObjectInput{
            Bucket: aws.String(bucket),
            Key:    aws.String(item),
        })
    if err != nil {
        fmt.Println(err)
    }

    fmt.Println("Downloaded", file.Name(), numBytes, "bytes")
}

This is the complete example from AWS Go SDK: https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/go/example_code/s3/s3_download_object.go

I wrote a blog post on how to test the S3 downloader, and created a testable S3 bucket scanner, which list all bucket's files and temporary download each file, see: https://medium.com/@tufin/a-testable-go-aws-s3-scanner-e54de0c26197

Here you can find the code: https://github.com/Tufin/blog/tree/master/s3-scanner