Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to find if a folder exists in hadoop or not?

I need to find if the input folder location exists or not in hadoop .

I am using the following command to do the same

 hadoop fs -test -d <folder Location> 

The query does not throw any error but also no output . I have checked it for both correct and incorrect location . What I understood from the documentation that it suppose to output 1 in case of correct location .

like image 808
pankaj jha Avatar asked Jul 28 '15 10:07

pankaj jha


People also ask

How do I find a folder in Hadoop?

Using the ls command, we can check for the directories in HDFS. Hadoop HDFS mkdir Command Description: This command creates the directory in HDFS if it does not already exist. Note: If the directory already exists in HDFS, then we will get an error message that file already exists.

How do you check if the HDFS directory is empty or not?

Solution. Use hdfs dfs -count to get the count of files and directories inside the directory. In this directory, we can see that the FILE_COUNT is 0 indicating the number of files is 0 in the directory.

How do I list only directories in Hadoop?

hadoop fs -ls -R command list all the files and directories in HDFS. grep “^d” will get you only the directories.

How do you check if a HDFS directory exists or not in Python?

isdir() Method to check if file exists. os. path. isdir() method in Python is used to check whether the specified path is an existing directory or not.


2 Answers

hdfs dfs -test -d <folder location> doesn't output anything, like 0 or 1. It's about exit status, 0 stands for a normal situation when the directory exists. 1 means a missing directory.

Here's an example you can use it in bash:

hdfs dfs -test -d /tmp && echo 'dir exists' || echo 'sorry, no such dir'
like image 193
Mikhail Golubtsov Avatar answered Nov 12 '22 05:11

Mikhail Golubtsov


thanks @Mikhail Golubtsov . Using the above hint my final modified shell script is

if hadoop fs -test -d  $1 ;
then echo "yeah it's there "
else
echo  "No its not there." 

fi
like image 45
pankaj jha Avatar answered Nov 12 '22 04:11

pankaj jha