Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rw-rw-rw- (on Windows)

Tags:

windows

hdfs

I am running Spark on Windows 7. When I use Hive, I see the following error

The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rw-rw-rw- 

The permissions are set as the following

C:\tmp>ls -la
total 20
drwxr-xr-x    1 ADMIN Administ        0 Dec 10 13:06 .
drwxr-xr-x    1 ADMIN Administ    28672 Dec 10 09:53 ..
drwxr-xr-x    2 ADMIN Administ        0 Dec 10 12:22 hive

I have set "full control" to all users from Windows->properties->security->Advanced.

But I still see the same error. Any help please? I have checked a bunch of links, some say this is a bug on Spark 1.5. Is this true?

Thanks Aarthi

like image 874
user1384205 Avatar asked Dec 10 '15 07:12

user1384205


6 Answers

First of all, make sure you are using correct Winutils for your OS. Then next step is permissions.
On Windows, you need to run following command on cmd:

D:\winutils\bin\winutils.exe chmod 777 D:\tmp\hive

Hope you have downloaded winutils already and set the HADOOP_HOME variable.

like image 146
Nishu Tayal Avatar answered Nov 16 '22 22:11

Nishu Tayal


First thing first check your computer domain. Try

c:\work\hadoop-2.2\bin\winutils.exe ls c:/tmp/hive

If this command says access denied or FindFileOwnerAndPermission error (1789): The trust relationship between this workstation and the primary domain failed.

It means your computer domain controller is not reachable , possible reason could be you are not on same VPN as your system domain controller.Connect to VPN and try again.

Now try the solution provided by Viktor or Nishu.

like image 38
Aaditya Raj Avatar answered Nov 16 '22 22:11

Aaditya Raj


You need to set this directory's permissions on HDFS, not your local filesystem. /tmp doesn't mean C:\tmp unless you set fs.defaultFs in core-site.xml to file://c:/, which is probably a bad idea.

Check it using

hdfs dfs -ls /tmp 

Set it using

hdfs dfs -chmod 777 /tmp/hive
like image 12
OneCricketeer Avatar answered Nov 16 '22 20:11

OneCricketeer


Next solution worked on Windows for me:

  • First, I defined HADOOP_HOME. It described in detail here
  • Next, I did like Nishu Tayal, but with one difference:C:\temp\hadoop\bin\winutils.exe chmod 777 \tmp\hive

\tmp\hive is not local directory

like image 11
L. Viktor Avatar answered Nov 16 '22 21:11

L. Viktor


Error while starting the spark-shell on VM running on Windows: Error msg: The root scratch dir: /tmp/hive on HDFS should be writable. Permission denied

Solution: /tmp/hive is temporary directory. Only temporary files are kept in this location. No problem even if we delete this directory, will be created when required with proper permissions.

Step 1) In hdfs, Remove the /tmp/hive directory ==> "hdfs dfs -rm -r /tmp/hive"

2) At OS level too, delete the dir /tmp/hive ==> rm -rf /tmp/hive

After this, started the spark-shell and it worked fine..

like image 9
SNK Avatar answered Nov 16 '22 22:11

SNK


This is a simple 4 step process:

For Spark 2.0+:

  1. Download Hadoop for Windows / Winutils
  2. Add this to your code (before SparkSession initialization):

    if(getOS()=="windows"){
        System.setProperty("hadoop.home.dir", "C:/Users//winutils-master/hadoop-2.7.1"); 
    }   
    
  3. Add this to your spark-session (You can change it to C:/Temp instead of Desktop).

    .config("hive.exec.scratchdir","C:/Users//Desktop/tmphive")
    
  4. Open cmd.exe and run:

    "path\to\hadoop-2.7.1\bin\winutils.exe" chmod 777 C:\Users\\Desktop\tmphive
    
like image 6
Abhinandan Dubey Avatar answered Nov 16 '22 21:11

Abhinandan Dubey