In a java app running on an edge node, I need to delete a hdfs folder, if it exists. I need to do that before running a mapreduce job (with spark) that output in the folder.
I found I could use the method
org.apache.hadoop.fs.FileUtil.fullyDelete(new File(url))
However, I can only make it work with local folder (i.e. file url on the running computer). I tried to use something like:
url = "hdfs://hdfshost:port/the/folder/to/delete";
with hdfs://hdfshost:port
being the hdfs namenode IPC. I use it for the mapreduce, so it is correct.
However it doesn't do anything.
So, what url should I use, or is there another method?
Note: here is the simple project in question.
Use an HDFS file manager to delete directories. See your Hadoop distribution's documentation to determine if it provides a file manager. Log into the Hadoop NameNode using the database administrator's account and use HDFS's rmr command to delete the directories.
rm: Remove a file from HDFS, similar to Unix rm command. This command does not delete directories. For recursive delete, use command -rm -r .
bin/hdfs dfs -mkdir /geeks => '/' means absolute path bin/hdfs dfs -mkdir geeks2 => Relative path -> the folder will be created relative to the home directory. touchz: It creates an empty file.
This works for me.
Just add the following codes in my WordCount program will do:
import org.apache.hadoop.fs.*;
...
Configuration conf = new Configuration();
Path output = new Path("/the/folder/to/delete");
FileSystem hdfs = FileSystem.get(URI.create("hdfs://namenode:port"),conf);
// delete existing directory
if (hdfs.exists(output)) {
hdfs.delete(output, true);
}
Job job = Job.getInstance(conf, "word count");
...
You need to add hdfs://hdfshost:port
explicitly to get distributed file system. Else the code will work for local file system only.
I do it this way:
Configuration conf = new Configuration();
conf.set("fs.hdfs.impl",org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
conf.set("fs.file.impl",org.apache.hadoop.fs.LocalFileSystem.class.getName());
FileSystem hdfs = FileSystem.get(URI.create("hdfs://<namenode-hostname>:<port>"), conf);
hdfs.delete("/path/to/your/file", isRecursive);
you don't need hdfs://hdfshost:port/
in your file path
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With