Spark - java IOException :Failed to create local dir in /tmp/blockmgr*

Question

I was trying to run a long running Spark Job. After few hours of execution, I get exception below :

Caused by: java.io.IOException: Failed to create local dir in /tmp/blockmgr-bb765fd4-361f-4ee4-a6ef-adc547d8d838/28 

Tried to get around it by checking:

1. Permission issue in /tmp dir. The spark server is not running as root. but /tmp dir should be writable to all users.

2. /tmp Dir has enough space.

Answer 1

Assuming that you are working with several nodes, you'll need to check every node participate in the spark operation (master/driver + slaves/nodes/workers).

Please confirm that each worker/node have enough disk space (especially check /tmp folder), and right permissions.

Answer 2

Edit: The answer below did not eventually solve my case. It's because some subfolders spark (or some of its dependencies) was able to create, yet not all of them. The frequent necessity of creation of such paths would make any project unviable. Therefore I ran Spark (PySpark in my case) as an Administrator, which solved the case. So in the end it is probably a permission issue afterall.

---

Original answer:
I solved the same problem I had on my local Windows machine (not a cluster). Since there was no problem with permissions, I created the dir that Spark was failing to create, i.e. I created the following folder as a local user and did not need to change any permissions on that folder.

C:\Users\<username>\AppData\Local\Temp\blockmgr-97439a5f-45b0-4257-a773-2b7650d17142