Java Copying File in HDFS to another Directory in HDFS

Question

I'm using the example in this link here to copy contents from one directory in hdfs to another directory in hdfs. The copying of file works, but it creates a new subdirectory in the target vs. just copying the file to the target dir. Example:

  Path source=new Path("hdfs://HANameService/sources/hpm_support/apc_code/");
  Path target=new Path("hdfs://HANameService/staging/hpm_support/apc_code/");
  FileSystem fs = source.getFileSystem(conf); 
  FileUtil.copy(fs, source, fs, target, true, conf);`

So instead of copying the file to hdfs://HANameService/staging/hpm_support/apc_code it creates a new dir under apc_code and the file ends up in hdfs://HANameService/staging/hpm_support/apc_code/apc_code How can I get it to not create that sub-directory?

Answer 1

You need to list the files in the source directory and copy each file using iterator

            Path source=new Path("hdfs://HANameService/sources/hpm_support/apc_code/");
            Path target=new Path("hdfs://HANameService/staging/hpm_support/apc_code/");
            FileSystem fs = source.getFileSystem(conf);
            RemoteIterator<LocatedFileStatus> sourceFiles = fs.listFiles(source, true);
            if(sourceFiles != null) {
                while(sourceFiles.hasNext()){
                    FileUtil.copy(fs, sourceFiles.next().getPath(), fs, target, true, conf);
                }           
            }

Hope it is helpful to you