could not delete files from dfs as safe mode is on

Question

I am using cloudera's VM to run hadoop. 99% of dfs space is used up. So I need to delete the files, but to delete the files the safe mode needs to be turned off.

I tried to turn off safe mode with the following commands,

sudo -u hdfs hdfs dfsadmin -safemode leave

It said its been turned off, but due to low space it automatically turns it on. And when I tried to delete file, it says since safe mode is on ,you cannot delete the files.

SafeModeException: Cannot create directory /user/cloudera/.Trash/Current. Name node is in safe mode. Resources are low on NN. Please add or free up more resources then turn off safe mode manually. NOTE: If you turn off safe mode before adding resources, the NN will immediately return to safe mode.. (error 403)

To delete files, safe mode needs to be turned off. To turn off safe mode, I need to free up space! So how do I clean up files to give more space?

Answer 1

You can try lowering the threshold for triggering safe mode by temporarily setting the following options to low values:

dfs.safemode.threshold.pct
dfs.namenode.safemode.threshold-pct

Also, the NameNode might not go back to safe mode immediately after leaving. In my case, I was able to clean up stuff by chaining the hdfs dfsadmin -safemode leave and hdfs dfs -rm commands together, e.g.

sudo su hdfs
hdfs dfsadmin -safemode leave; hdfs dfs -rm -skipTrash /path/to/stuff/to/delete