Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Checking if directory in HDFS already exists or not

I am having following directory structure in HDFS,

/analysis/alertData/logs/YEAR/MONTH/DATE/HOURS

That is data is coming on houly basis and stored in format of year/month/day/hour.

I have written a shell script in which i am passing path till

"/analysis/alertData/logs"   ( this will vary depending on what product of data i am handling)

then shell script go through the year/month/date/hour folders and return the most latest path.

For example:

 Directories present in HDFS has following structure: 

 /analysis/alertData/logs/2014/10/22/01
 /analysis/alertData/logs/2013/5/14/04

 shell script is given path till :   " /analysis/alertData/logs "

 it outputs most recent directory :    /analysis/alertData/logs/2014/10/22/01

My question is here is how can i validate whether HDFS directory path pass to shell script is valid or not. Lets say i pass a wrong path as input or path which does not exist so how to handle that in shell script.

Sample wrong path can be:

  wrong path   :  /analysis/alertData ( correct path :  /analysis/alertData/logs/ )
  wrong path   :  /abc/xyz/  ( path does not exit in HDFS )

I tried using Hadoop dfs -test -z/-d/-e options did not worked for me. Any suggestion for this.

NOTE : Not posting my original code here, as solution to my problem does not depend on it.

Thanks in advance.

like image 436
user1188611 Avatar asked Oct 22 '14 17:10

user1188611


People also ask

How do you check if the hdfs directory is empty or not?

Solution. Use hdfs dfs -count to get the count of files and directories inside the directory. In this directory, we can see that the FILE_COUNT is 0 indicating the number of files is 0 in the directory.

How do I find my hdfs directory?

If you type hdfs dfs -ls / you will get list of directories in hdfs.


4 Answers

Try w/o test command []:

if $(hadoop fs -test -d $yourdir) ; then echo "ok";else echo "not ok"; fi
like image 135
morsik Avatar answered Oct 14 '22 05:10

morsik


Since

hdfs dfs -test -d $yourdir

return 0 if exists, then

if [ $? == 0 ]; then
    echo "exists"
else
    echo "dir does not exists"
fi
like image 34
silvio.digregorio Avatar answered Oct 14 '22 04:10

silvio.digregorio


Hadoop fs is deprecated Usage: hdfs dfs -test -[ezd] URI

Options: The -e option will check to see if the file exists, returning 0 if true. The -z option will check to see if the file is zero length, returning 0 if true. The -d option will check to see if the path is directory, returning 0 if true. Example: hdfs dfs -test -d $yourdir

Please check the following for more info: https://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/FileSystemShell.html Regards

like image 6
Ngoc Nguyen Avatar answered Oct 14 '22 03:10

Ngoc Nguyen


Hi I have used following script to test the HDFS directory exists or not. I have seen in your question that you tried this test command and not worked. Could you please provide any trace on why this not working..

 hadoop fs -test -d $dirpath
    if [ $? != 0 ]
            then
                hadoop fs -mkdir $dirpath
                else
                    echo "Directory already present in HDFS"
    fi
like image 5
scalauser Avatar answered Oct 14 '22 03:10

scalauser