I want to move files around in HDFS using the Java APIs. I cannot figure out a way to do this. The FileSystem class only seems to want to allow moving to and from the local file system.. but I want to keep them in HDFS and move them there.
Am I missing something basic? The only way I can figure to do it is to read it from the input stream and write it back out... and then delete the old copy (yuck).
thanks
Use FileSystem.rename():
public abstract boolean rename(Path src, Path dst) throws IOException
Renames Path
src
to Pathdst
. Can take place on local fs or remote DFS.Parameters:
src
- path to be renameddst
- new path after rename
Returns:true
if rename is successful
Throws:
IOException - on failure
The java.nio.* approach may not work on HDFS always. So found the following solution that works.
Move files from one directory to another using org.apache.hadoop.fs.FileUtil.copy API
val fs = FileSystem.get(new Configuration())
val conf = new org.apache.hadoop.conf.Configuration()
val srcFs = FileSystem.get(new org.apache.hadoop.conf.Configuration())
val dstFs = FileSystem.get(new org.apache.hadoop.conf.Configuration())
val dstPath = new org.apache.hadoop.fs.Path(DEST_FILE_DIR)
for (file <- fileList) {
// The 5th parameter indicates whether source should be deleted or not
FileUtil.copy(srcFs, file, dstFs, dstPath, true, conf)
I think the FileUtilts replaceFile would also solve the purpose. http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/FileUtil.html#replaceFile(java.io.File, java.io.File)
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With