Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Permission denied at hdfs

I am new to hadoop distributed file system, I have done complete installation of hadoop single node on my machine.but after that when i am going to upload data to hdfs it give an error message Permission Denied.

Message from terminal with command:

hduser@ubuntu:/usr/local/hadoop$ hadoop fs -put /usr/local/input-data/ /input
put: /usr/local/input-data (Permission denied)

hduser@ubuntu:/usr/local/hadoop$ 

After using sudo and adding hduser to sudouser:

hduser@ubuntu:/usr/local/hadoop$ sudo bin/hadoop fs -put /usr/local/input-data/ /inwe
put: org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="":hduser:supergroup:rwxr-xr-x

hduser@ubuntu:/usr/local/hadoop$ 
like image 666
Vignesh Prajapati Avatar asked Jul 21 '12 15:07

Vignesh Prajapati


People also ask

How do I change ownership in HDFS?

Changing the owner of files in the HDFS: Firstly, switch to root user from ec2-user using the “sudo -i” command. And let us create a directory in the HDFS by changing it as the HDFS user.

Where is my HDFS directory?

If you type hdfs dfs -ls / you will get list of directories in hdfs.


3 Answers

I solved this problem temporary by disabling the dfs permission.By adding below property code to conf/hdfs-site.xml

<property>
  <name>dfs.permissions</name>
  <value>false</value>
</property>
like image 91
Vignesh Prajapati Avatar answered Oct 25 '22 01:10

Vignesh Prajapati


I had similar situation and here is my approach which is somewhat different:

 HADOOP_USER_NAME=hdfs hdfs dfs -put /root/MyHadoop/file1.txt /

What you actually do is you read local file in accordance to your local permissions but when placing file on HDFS you are authenticated like user hdfs. You can do this with other ID (beware of real auth schemes configuration but this is usually not a case).

Advantages:

  1. Permissions are kept on HDFS.
  2. You don't need sudo.
  3. You don't need actually appropriate local user 'hdfs' at all.
  4. You don't need to copy anything or change permissions because of previous points.
like image 56
Roman Nikitchenko Avatar answered Oct 25 '22 02:10

Roman Nikitchenko


You are experiencing two separate problems here:


hduser@ubuntu:/usr/local/hadoop$ hadoop fs -put /usr/local/input-data/ /input put: /usr/local/input-data (Permission denied)

Here, the user hduser does not have access to the local directory /usr/local/input-data. That is, your local permissions are too restrictive. You should change it.


hduser@ubuntu:/usr/local/hadoop$ sudo bin/hadoop fs -put /usr/local/input-data/ /inwe put: org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="":hduser:supergroup:rwxr-xr-x

Here, the user root (since you are using sudo) does not have access to the HDFS directory /input. As you can see: hduser:supergroup:rwxr-xr-x says only hduser has write access. Hadoop doesn't really respect root as a special user.


To fix this, I suggest you change the permissions on the local data:

sudo chmod -R og+rx /usr/local/input-data/

Then, try the put command again as hduser.

like image 18
Donald Miner Avatar answered Oct 25 '22 00:10

Donald Miner