Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to upload folder with subfolder to amazon s3?

I need upload folder with subfolders on amazon s3. I try upload with this snipet.

    for (Path path : directoryWalk("/home/rmuhamedgaliev/tmp/eota7tas0cdlg2ufq5mlke7olf/")){
       if (!path.getParent().toString().equals("eota7tas0cdlg2ufq5mlke7olf")){
        amazonS3Client.putObject("*****", "/plans/eota7tas0cdlg2ufq5mlke7olf/" + path.getParent().toString() + "/" + path.getFileName(), new File(path.toString()));
       } else {
        amazonS3Client.putObject("*******", "/plans/eota7tas0cdlg2ufq5mlke7olf/" + path.getFileName(), new File(path.toString()));
       }
    }

But this code create full path files with ("/home/rmuhamedgaliev/tmp/eota7tas0cdlg2ufq5mlke7olf"). How to upload it with path ("/plans/eota7tas0cdlg2ufq5mlke7olf/{subfolders and files}")

    private List<Path> directoryWalk(String path) throws IOException {
        final List<Path> files = new ArrayList<>();
        Files.walkFileTree(Paths.get(path), new SimpleFileVisitor<Path>() {

            @Override
            public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
                files.add(file);
                return FileVisitResult.CONTINUE;
            }
        });
        return files;
    }
like image 204
Rinat Mukhamedgaliev Avatar asked Feb 26 '13 13:02

Rinat Mukhamedgaliev


People also ask

Can a user upload a whole folder and subfolders to S3 bucket in a single click using Amazon S3 console?

To upload folders and files to an S3 bucketSign in to the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3/ . In the Buckets list, choose the name of the bucket that you want to upload your folders or files to. Choose Upload.

Can S3 have folders?

In Amazon S3, folders are used to group objects and organize files. Unlike a traditional file system, Amazon S3 doesn't use hierarchy to organize its objects and files. Amazon S3 console supports the folder concept only as a means of grouping (and displaying) objects.

Is it better to have multiple S3 buckets or one bucket with sub folders?

there's a performance difference depending on how your spread your keys in S3. Said performance difference applies the same to buckets as it does to keys. In the context of this question, there is no difference.


2 Answers

Have you looked at the TransferManager in the AWS SDK for Java? You could use the uploadDirectory method for this. The javadoc is here. In essence, you could do something like this:

transferManager.uploadDirectory(bucketName, "plans/eota7tas0cdlg2ufq5mlke7olf/", new File("/home/rmuhamedgaliev/tmp/eota7tas0cdlg2ufq5mlke7olf/"), true);
like image 155
Wade Matveyenko Avatar answered Sep 26 '22 11:09

Wade Matveyenko


I writen my own way.

        List<File> files = new LinkedList<File>();
        listFiles(new File("/home/rmuhamedgaliev/tmp/eota7tas0cdlg2ufq5mlke7olf"), files, true);
        for (File f : files) {
            String key = f.getAbsolutePath().substring(new File("/home/rmuhamedgaliev/tmp/eota7tas0cdlg2ufq5mlke7olf").getAbsolutePath().length() + 1)
                .replaceAll("\\\\", "/");
            amazonS3Client.putObject("****", "plans/eota7tas0cdlg2ufq5mlke7olf/" + key, f);
        }
like image 31
Rinat Mukhamedgaliev Avatar answered Sep 24 '22 11:09

Rinat Mukhamedgaliev