Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

HDFS Home Directory

I have setup a single node multi-user hadoop cluster. In my cluster, there is an admin user that is responsible for running the cluster (superuser). All other users are allocated a hdfs directory like /home/xyz where xyz is a username.

In unix, we can change the default home directory for a user in /etc/passwd. And by default, landing directory for a user is the home directory.

How do I do it in hadoop for hdfs file system. I want for example, if user types: $hadoop dfs -ls on the unix prompt. It shall list the contents of the home directory allocated by me.

Further, hdfs directories are created by the superuser who runs the cluster(hadoop superuser and not unix root) and then transfers the ownership to a particular user.

like image 658
Tapan Avasthi Avatar asked Apr 09 '12 06:04

Tapan Avasthi


People also ask

How do I find my hdfs home directory?

If you type hdfs dfs -ls / you will get list of directories in hdfs. Then you can transfer files from local file system to hdfs using -copyFromLocal or -put to a particular directory or using -mkdir you can create new directory.

How do I go to a directory in hdfs?

There is no cd (change directory) command in hdfs file system. You can only list the directories and use them for reaching the next directory. You have to navigate manually by providing the complete path using the ls command.

What is hdfs directory?

In Hadoop, both the input and output of a job are usually stored in a shared file system called the Hadoop Distributed File System (HDFS). As its name implies, HDFS is a file system that is distributed across the nodes of a cluster, and that provides a unified interface to the distributed files.


1 Answers

I'm not sure this is something that can be configured - the source for DistributedFileSystem(line 150) has a call for getHomeDirectory that seems to be hard-coded:

@Override
public Path getHomeDirectory() {
  return makeQualified(new Path("/user/" + dfs.ugi.getShortUserName()));
}

You do have two possible choices if you want to be able to change this:

  • Submit a ticket to hadoop asking for a new feature - See this link
  • Amend the source yourself and re-build + re-distribute the hadoop-core jar across your cluster (simple in your single node pseudo cluster)
like image 81
Chris White Avatar answered Oct 12 '22 08:10

Chris White