shutil.rmtree: FileNotFoundError: [Errno 2] No such file or directory: '._xxx'

Question

Code and error (Reduced, if you'd like to see the full code, refer to the last section):

import shutil
import os

filteredCleaned = 
 '/Volumes/Extreme_SSD/Raymond_Lab/Day_4_Rotarod_Videos_Rotated_if_Necessary_copy/filtered_cleaned_WT_cleaned_YAC128'

if os.path.exists(filteredCleaned):
    shutil.rmtree(filteredCleaned)

^ Showing hidden files (pressed shift+command+.) and other dirs are showing hidden files if exist.

runfile('/Users/ksb7640/Documents/UBC_Academic/Raymond_Lab/448/rotarod/svm_all/data_filter.py', wdir='/Users/ksb7640/Documents/UBC_Academic/Raymond_Lab/448/rotarod/svm_all')
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3417, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-b9676f9bf96c>", line 1, in <module>
    runfile('/Users/ksb7640/Documents/UBC_Academic/Raymond_Lab/448/rotarod/svm_all/data_filter.py', wdir='/Users/ksb7640/Documents/UBC_Academic/Raymond_Lab/448/rotarod/svm_all')
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/ksb7640/Documents/UBC_Academic/Raymond_Lab/448/rotarod/svm_all/data_filter.py", line 21, in <module>
    shutil.rmtree(filteredCleaned)
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/shutil.py", line 715, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/shutil.py", line 672, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/shutil.py", line 670, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
FileNotFoundError: [Errno 2] No such file or directory: '._filtered_combined_Experiment2-190630_Day4_145m2_rotarod2_Cam2onRotarodDeepCut_resnet50_rotarod3Jul17shuffle1_1030000.csv'

Question:

Where did '._filtered_combined_Experiment2-190630_Day4_145m2_rotarod2_Cam2onRotarodDeepCut_resnet50_rotarod3Jul17shuffle1_1030000.csv' come from?

I did use df.to_csv(csv, index=False) to create filtered_combined_Experiment2-190630_Day4_145m2_rotarod2_Cam2onRotarodDeepCut_resnet50_rotarod3Jul17shuffle1_1030000.csv in filtered_cleaned_WT_cleaned_YAC128 but I never used chmod to hide them.

Also, as you can see in the screenshot, there are no even hidden files. What is the error about?

Full code that might be REDUNDANT, but just in case... (for relevent, shortened code, see the top.)

data_filter.py

import shutil

from export_df_to_csv import export_df_to_csv
from extract_parent_current import extract_parent_current
from import_df import *

prefix = 'filtered_'

bound = 0.9

# make a directory for output files
cleanedWT = '/Volumes/Extreme_SSD/Raymond_Lab/Day_4_Rotarod_Videos_Rotated_if_Necessary_copy/cleaned_WT'
cleanedYAC128 = '/Volumes/Extreme_SSD/Raymond_Lab/Day_4_Rotarod_Videos_Rotated_if_Necessary_copy/cleaned_YAC128'

cleanedWTParentDir, WTdir = extract_parent_current(cleanedWT)
cleanedYAC128ParentDir, YAC128dir = extract_parent_current(cleanedYAC128)

filteredCleaned = os.path.join(cleanedYAC128ParentDir, prefix + WTdir + '_' + YAC128dir)

if os.path.exists(filteredCleaned):
    shutil.rmtree(filteredCleaned)
os.mkdir(filteredCleaned)

csvs_labels_arr = import_csvs(cleanedWT, cleanedYAC128)
paths_dfs_labels_arr = csvs_to_paths_dfs_labels_arr(csvs_labels_arr)
for path_df_label in paths_dfs_labels_arr:
    _, dfFileName = extract_parent_current(path_df_label[0])
    df = path_df_label[1]

    # filter out rows below bound
    df = df[
        (df['Rightpaw likelihood'] > bound) & (df['Leftpaw likelihood'] > bound) & (df['Tail likelihood'] > bound)]

    export_df_to_csv(df, os.path.join(filteredCleaned, prefix + dfFileName))

export_df_to_csv.py

input_file = '/Volumes/Extreme SSD/Raymond Lab/Day_4_Rotarod_Videos_Rotated_if_Necessary copy/cleaned_WT/cleaned_Experiment2-190630_Day4_145m1_rotarod3_Cam2onRotarodDeepCut_resnet50_rotarod3Jul17shuffle1_1030000.csv'
output_file = '/Volumes/Extreme SSD/Raymond Lab/Day_4_Rotarod_Videos_Rotated_if_Necessary copy/cleaned_WT/cleaned_Experiment2-190630_Day4_145m1_rotarod3_Cam2onRotarodDeepCut_resnet50_rotarod3Jul17shuffle1_1030000.csv'


def export_df_to_csv(df, csv):
    df.to_csv(csv, index=False)

extract_parent_current.py

import os


def extract_parent_current(dir):
    if str.endswith(dir, '/'):
        dir = dir[:-1]
    return os.path.split(dir)

import_df.py

import os
import random
from copy import deepcopy
import pandas as pd


def import_csvs(WT_file_path, YAC_file_path):
    csv_paths_arr = []
    for root, dirs, files in os.walk(WT_file_path, topdown=False):
        for file in files:
            if not file.startswith('.'):
                csv_paths_arr.append([os.path.join(root, file), 0])

    for root, dirs, files in os.walk(YAC_file_path, topdown=False):
        for file in files:
            if not file.startswith('.'):
                csv_paths_arr.append([os.path.join(root, file), 1])

    return csv_paths_arr


def csvs_to_paths_dfs_labels_arr(csvpaths_labels_arr):
    paths_dfs_labels_arr = deepcopy(csvpaths_labels_arr)
    for i, csvpath_label_arr in enumerate(csvpaths_labels_arr):
        (paths_dfs_labels_arr[i])[0] = pd.read_csv(csvpath_label_arr[0], encoding='unicode_escape')
        paths_dfs_labels_arr[i].insert(0, csvpath_label_arr[0])
    return paths_dfs_labels_arr


def import_df(WT_file_path, YAC_file_path):
    csv_paths_arr = import_csvs(WT_file_path, YAC_file_path)
    dfs_labels = csvs_to_paths_dfs_labels_arr(csv_paths_arr)
    random.shuffle(dfs_labels)
    return dfs_labels

Answer 1

UPDATE: Underlying python bug has been fixed by cpython#14064, which will be part of python 3.13 (mitigation code for earlier versions below).

I already provided this answer for the linked question Delete directory and all symlinks recursively, but decided to reproduce it here. Over there it falls into the fallacy to provide a solution to the underlying problem rather than an answer to the actual question, while it here directly answers where ._filtered_combined_Experiment2-190630_Day4_145m2_rotarod2_Cam2onRotarodDeepCut_resnet50_rotarod3Jul17shuffle1_1030000.csv comes from.

The /Volumes/Extreme_SSD/ path suggests you are on Mac OSX and your directory is at least partially on a non-Mac filesystem (ie not HFS+). On those, Mac filesystem drivers automatically create binary companion files prefixed with ._ to record so-called extended attributes (explained in https://apple.stackexchange.com/questions/14980/why-are-dot-underscore-files-created-and-how-can-i-avoid-them, but also illustrated below), during mundane operations like looking at the files with default OSX viewers.

rmtree on systems which do not support file descriptors in os.scandir (like Mac OSX) now unsafely creates a list of entries and then unlinks them one by one (creating a known race-condition: https://github.com/python/cpython/blob/908fd691f96403a3c30d85c17dd74ed1f26a60fd/Lib/shutil.py#L592-L621). Unfortunately two separate behaviours make this condition true every time:

1. the original file is always listed before the extended attributes one, and
2. when the original file is unlinked (test.txt) the meta file (._test.txt) is removed simultaneously.

Thus, the extended attribute file will be missing when it is its turn and throw the FileNotFoundError you are experiencing.

I think this bug would be best addressed by cpython#14064, which aims at ignoring FileNotFoundErrors in rmtree generally.

Mitigation

In the mean time you could ignore unlinking errors on those meta files with onerror:

def ignore_extended_attributes(func, filename, exc_info):
    is_meta_file = os.path.basename(filename).startswith("._")
    if not (func is os.unlink and is_meta_file):
        raise

shutil.rmtree(path_dir, onerror=ignore_extended_attributes)

Show case of Mac's extended attributes

To illustrate you can create a small ExFAT disk image and mount it to /Volumes/Untitled with the commands

hdiutil create -size 5m -fs exfat test.dmg
hdiutil attach test.dmg            # mounts at /Volumes/Untitled
cd /Volumes/Untitled

mkdir test                         # create a directory to remove
cd test
touch test.txt
open test.txt                      # open the test.txt file in the standard editor 

Just opening the file in the standard text editor creates an extended attributes file ._test.txt and records the last access time in it:

/Volumes/Untitled/test $ ls -a
.          ..         ._test.txt test.txt
/Volumes/Untitled/test $ xattr test.txt
com.apple.lastuseddate#PS

The problem is that unlinking the original file automatically also unlinks the companion file.

/Volumes/Untitled/test $ rm test.txt
/Volumes/Untitled/test $ ls -a
.          ..