I need to find if the input folder location exists or not in hadoop .
I am using the following command to do the same
hadoop fs -test -d <folder Location>
The query does not throw any error but also no output . I have checked it for both correct and incorrect location . What I understood from the documentation that it suppose to output 1 in case of correct location .
Using the ls command, we can check for the directories in HDFS. Hadoop HDFS mkdir Command Description: This command creates the directory in HDFS if it does not already exist. Note: If the directory already exists in HDFS, then we will get an error message that file already exists.
Solution. Use hdfs dfs -count to get the count of files and directories inside the directory. In this directory, we can see that the FILE_COUNT is 0 indicating the number of files is 0 in the directory.
hadoop fs -ls -R command list all the files and directories in HDFS. grep “^d” will get you only the directories.
isdir() Method to check if file exists. os. path. isdir() method in Python is used to check whether the specified path is an existing directory or not.
hdfs dfs -test -d <folder location>
doesn't output anything, like 0
or 1
. It's about exit status, 0
stands for a normal situation when the directory exists. 1
means a missing directory.
Here's an example you can use it in bash:
hdfs dfs -test -d /tmp && echo 'dir exists' || echo 'sorry, no such dir'
thanks @Mikhail Golubtsov . Using the above hint my final modified shell script is
if hadoop fs -test -d $1 ;
then echo "yeah it's there "
else
echo "No its not there."
fi
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With