Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

AWS S3 uploaded files not showing

I am using golang sdk to upload files to a bucket on amazon S3.The response gives no error and the file is uploaded successfully. My problem is when I am listing the objects of the bucket in the same region in which I have uploaded a new .jpg file. I am getting same files list with no new file added to the Bucket. I don't know what I am doing wrong. This is the code that I am using to list objects inside the bucket.

input := &s3.ListObjectsInput{
    Bucket: aws.String("Bucket Name"),
}
result2, err := svc.ListObjects(input)
if err != nil {
    if aerr, ok := err.(awserr.Error); ok {
        switch aerr.Code() {
        case s3.ErrCodeNoSuchBucket:
            fmt.Println(s3.ErrCodeNoSuchBucket, aerr.Error())
        default:
            fmt.Println(aerr.Error())
        }
    } else {
        // Print the error, cast err to awserr.Error to get the Code and
        // Message from an error.
        fmt.Println(err.Error())
    }
    return
}
log.Println("Bucket List", result2)
like image 825
Himanshu Avatar asked Dec 05 '22 11:12

Himanshu


1 Answers

s3 is eventually consistent and may not show an object that has been recently written in a list, even from within the same zone:

Amazon S3 achieves high availability by replicating data across multiple servers within Amazon's data centers. If a PUT request is successful, your data is safely stored. However, information about the changes must replicate across Amazon S3, which can take some time, and so you might observe the following behaviors:

A process writes a new object to Amazon S3 and immediately lists keys within its bucket. Until the change is fully propagated, the object might not appear in the list.

A process replaces an existing object and immediately attempts to read it. Until the change is fully propagated, Amazon S3 might return the prior data.

A process deletes an existing object and immediately attempts to read it. Until the deletion is fully propagated, Amazon S3 might return the deleted data.

A process deletes an existing object and immediately lists keys within its bucket. Until the deletion is fully propagated, Amazon S3 might list the deleted object.

see S3 developer guide

There is a polling Go AWS API call "func (*S3) WaitUntilObjectExists" which keeps looking for a file until it appears or a time out happens

Update s3 is now strongly consistent thanks to @Tensiba for pointing this out!

Amazon S3 provides strong read-after-write consistency for PUT and DELETE requests of objects in your Amazon S3 bucket in all AWS Regions. This behavior applies to both writes to new objects as well as PUT requests that overwrite existing objects and DELETE requests. In addition, read operations on Amazon S3 Select, Amazon S3 access controls lists (ACLs), Amazon S3 Object Tags, and object metadata (for example, the HEAD object) are strongly consistent.

like image 55
Vorsprung Avatar answered Dec 09 '22 16:12

Vorsprung