Repeated data.table fread and fwrite causes "Permission denied" error

Question

I encountered this issue using the data.table fwrite() and fread() functions for managing resources in a parallel calculation, but was also able to recreate the behavior in the below sequential example code. Calling fwrite() throws the following error:

Error in fwrite(dt, csv_path) : Permission denied: 'D:/mypath/test.csv'. Failed to open existing file for writing. Do you have write permission to it? Is this Windows and does another process such as Excel have it open?

The behavior seems to be related to the calling of fread() right before, as commenting out the fread() command makes the error disappear. Depending on your system, you might have to increase the number of iterations before the error occurs as it occurs at varying iteration numbers.

Does anyone have an idea why this is happening? Thanks in advance for your assistance!

Example code:

library(data.table)

dt = data.table(a = c(1, 2), b = c("a", "b"))
csv_path = "D:/mypath/test.csv"
fwrite(dt, csv_path)

for(i in 1:10000){
  test = fread(csv_path)
  fwrite(dt, csv_path)
}

System info

R version 4.0.0 (2020-04-24)

Platform: x86_64-w64-mingw32/x64 (64-bit)

Running under: Windows Server x64 (build 14393)

data.table_1.12.8

Answer 1

I tried your code on a Windows machine and I was not able reproduce it.

I believe the issue is related to Windows file handler, which seems to be not fast enought to close file connection before opening it again.

You can try following code to see if it is reproducible just in R:

x = "a,b\n1,a\n2,b\n"

csv_path = "D:/mypath/test.csv"
file.create(csv_path)
f = file(csv_path, "w")
cat(x, file=f)
close(f)

for (i in 1:10000) {
  f = file(csv_path, "r")
  test = readLines(f)
  close(f)
  f = file(csv_path, "w")
  cat(x, file=f)
  close(f)
}

What could also make sense is it see how much Sys.sleep is enough to make the problem disappear.