Delete objects in s3 using wildcard matching

Question

I have the following working code to delete an object from Amazon s3

params := &s3.DeleteObjectInput{
        Bucket: aws.String("Bucketname"),
        Key : aws.String("ObjectKey"),
    }
s3Conn.DeleteObjects(params)

But what i want to do is to delete all files under a folder using wildcard **. I know amazon s3 doesn't treat "x/y/file.jpg" as a folder y inside x but what i want to achieve is by mentioning "x/y*" delete all the subsequent objects having the same prefix. Tried amazon multi object delete

params := &s3.DeleteObjectsInput{
        Bucket: aws.String("BucketName"),
        Delete: &s3.Delete{
            Objects: []*s3.ObjectIdentifier {
                {
                    Key : aws.String("x/y/.*"), 
                },
            },
        },
    }
    result , err := s3Conn.DeleteObjects(params)

I know in php it can be done easily by s3->delete_all_objects as per this answer. Is the same action possible in GOlang.

Answer 1

Unfortunately the goamz package doesn't have a method similar to the PHP library's delete_all_objects.

However, the source code for the PHP delete_all_objects is available here (toggle source view): http://docs.aws.amazon.com/AWSSDKforPHP/latest/#m=AmazonS3/delete_all_objects

Here are the important lines of code:

public function delete_all_objects($bucket, $pcre = self::PCRE_ALL)
{
// Collect all matches
    $list = $this->get_object_list($bucket, array('pcre' => $pcre));

    // As long as we have at least one match...
    if (count($list) > 0)
    {
        $objects = array();

        foreach ($list as $object)
        {
            $objects[] = array('key' => $object);
        }

        $batch = new CFBatchRequest();
        $batch->use_credentials($this->credentials);

        foreach (array_chunk($objects, 1000) as $object_set)
        {
            $this->batch($batch)->delete_objects($bucket, array(
                'objects' => $object_set
            ));
        }

        $responses = $this->batch($batch)->send();

As you can see, the PHP code will actually make an HTTP request on the bucket to first get all files matching PCRE_ALL, which is defined elsewhere as const PCRE_ALL = '/.*/i';.

You can only delete 1000 files at once, so delete_all_objects then creates a batch function to delete 1000 files at a time.

You have to create the same functionality in your go program as the goamz package doesn't support this yet. Luckily it should only be a few lines of code, and you have a guide from the PHP library.

It might be worth submitting a pull request for the goamz package once you're done!

Answer 2

Using the mc tool you can do:

mc rm -r --force https://BucketName.s3.amazonaws.com/x/y

it will delete all the objects with the prefix "x/y"

You can achieve the same with Go using minio-go like this:

package main

import (
    "log"

    "github.com/minio/minio-go"
)

func main() {
    config := minio.Config{
        AccessKeyID:     "YOUR-ACCESS-KEY-HERE",
        SecretAccessKey: "YOUR-PASSWORD-HERE",
        Endpoint:        "https://s3.amazonaws.com",
    }
    // find Your S3 endpoint here http://docs.aws.amazon.com/general/latest/gr/rande.html

    s3Client, err := minio.New(config)
    if err != nil {
        log.Fatalln(err)
    }
    isRecursive := true
    for object := range s3Client.ListObjects("BucketName", "x/y", isRecursive) {
        if object.Err != nil {
            log.Fatalln(object.Err)
        }
        err := s3Client.RemoveObject("BucketName", object.Key)
        if err != nil {
            log.Fatalln(err)
            continue
        }
        log.Println("Removed : " + object.Key)
    }
}