Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Opening a file stored in HDFS to edit in VI

I would like to edit a text file directly in HDFS using VI without having to copy it to local, edit it and then copy it back from local. Is this possible?

Edit: This used to be possible in Cloudera's Hue UI but is no longer the case.

like image 600
Michal Avatar asked Nov 07 '14 19:11

Michal


People also ask

How do I edit a file in HDFS?

You can not modified data once stored in hdfs because hdfs follows Write Once Read Many model. You can only append the data once stored in hdfs.

How do I open an HDFS file?

You can use the Hadoop filesystem command to read any file. It supports the cat command to read the content.

Is that possible to edit the file present in the HDFS?

In HDFS we cannot edit the files which are already stored in HDFS, but we can append data by reopening the files.


2 Answers

There are couple of options that you could try, which allows you to mount HDFS to your local machine and then you could use your local system commands like cp, rm, cat, mv, mkdir, rmdir, more, etc. But neither of them supports random write operations but supports append operations.

  • NFS Gateway
  • Hadoop Fuse

NFS Gateway uses NFS V3 and support appending to file but could not perform random write operations.

And regarding your comment on hue, maybe Hue is downloading the file to a local buffer and after editing it might be replacing the original file in HDFS.

like image 131
Ashrith Avatar answered Sep 20 '22 10:09

Ashrith


A simple way is to copy from and to hdfs, and edit locally (See here)

hvim <filename>

Source code of hvim

hadoop fs -text $1>hvim.txt
vim hvim.txt
hadoop fs -rm -skipTrash $1
hadoop fs -copyFromLocal hvim.txt $1
rm hvim.txt
like image 42
Uri Goren Avatar answered Sep 18 '22 10:09

Uri Goren