Menu
I want to copy file in THE SAME HDFS ,just like copy file from HDFS://abc:9000/user/a.txt to HDFS://abc:9000/user/123/
Can I do that by using JAVA API? Thanks
267
The Hadoop fs shell command – put is used to copy the file from local file system to Hadoop HDFS file system. similarly HDFS also has – copyFromLocal. Below is the usage of -put command.
Writing A File To HDFS – Java Program Writing a file to HDFS is very easy, we can simply execute hadoop fs -copyFromLocal command to copy a file from local filesystem to HDFS. In this post we will write our own Java program to write the file from local file system to HDFS. Here is the program – FileWriteToHDFS.java
HDFS is a distributed file system for storing very large data files, running on clusters of commodity hardware. It is fault tolerant, scalable, and extremely simple to expand. Hadoop comes bundled with HDFS ( Hadoop Distributed File Systems ).
If the file already exists on HDFS, you will get an error message saying “File already exists”. In order to overwrite the file use -f option. Note that you can use it with either hadoop fs -put or hdfs dfs -put to upload files from the local file system to HDFS, both return the same results.
FileUtil provides a method for copying files.
Configuration configuration = new Configuration();
configuration.set("fs.defaultFS", "hdfs://abc:9000");
FileSystem filesystem = FileSystem.get(configuration);
FileUtil.copy(filesystem, new Path("src/path"), filesystem, new Path("dst/path"), false, configuration);
If you need to copy it to another cluster, just make a new Configuration and setup and new FileSystem.
179
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
