"WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41" during "conda install"

Question

Starting today I get a lot of

WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41

warnings when I try to update or install packages using conda install or conda update. For example:

(...) C:\Users\...> conda install numba
Fetching package metadata ...........
Solving package specifications: .

Package plan for installation in environment C:\...:

The following packages will be DOWNGRADED due to dependency conflicts:

    numba: 0.30.0-np111py35_0 --> 0.30.1-np111py35_0

Proceed ([y]/n)? y

numba-0.30.0-np111p35_0 100% |###############################| Time: 0:00:00   2.50 MB/s
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno 41

The packages are installed afterwards but all these Warnings seem to indicate that something doesn't work properly.

OS: Windows 10 64 bit
conda: 4.3.4

Could you tell what I need to do to fix these Warnings? Or can I ignore them?

Answer 1

These warnings that you see are "normal" according to the current conda source tree. To understand the origin of the aforementioned warnings, let's have a look at the source code in questions and a recent commit in the conda repository (https://github.com/conda/conda). The relevant source code that prints the warnings you see is the following:

https://github.com/conda/conda/blob/4.3.4/conda/gateways/disk/init.py

# -*- coding: utf-8 -*-
from __future__ import absolute_import, division, print_function, unicode_literals

import sys
from errno import EACCES, ENOENT, EPERM, EPROTOTYPE
from logging import getLogger
from os.path import basename
from time import sleep

from ...common.compat import on_win

log = getLogger(__name__)

MAX_TRIES = 7


def exp_backoff_fn(fn, *args, **kwargs):
    """Mostly for retrying file operations that fail on Windows due to virus scanners"""
    max_tries = kwargs.pop('max_tries', MAX_TRIES)
    if not on_win:
        return fn(*args, **kwargs)

    import random
    # with max_tries = 6, max total time ~= 3.2 sec
    # with max_tries = 7, max total time ~= 6.5 sec
    for n in range(max_tries):
        try:
            result = fn(*args, **kwargs)
        except (OSError, IOError) as e:
            log.trace(repr(e))
            if e.errno in (EPERM, EACCES):
                if n == max_tries-1:
                    raise
                sleep_time = ((2 ** n) + random.random()) * 0.1
                caller_frame = sys._getframe(1)
                log.trace("retrying %s/%s %s() in %g sec",
                          basename(caller_frame.f_code.co_filename),
                          caller_frame.f_lineno,
                          fn.__name__,
                          sleep_time)
                sleep(sleep_time)
            elif e.errno in (ENOENT, EPROTOTYPE):
                # errno.ENOENT File not found error / No such file or directory
                # errno.EPROTOTYPE OSError(41, 'The directory is not empty')
                raise
            else:
                log.warn("Uncaught backoff with errno %d", e.errno)
                raise
        else:
            return result

From the above source code it appears to be a warning thrown on Windows that might appear when

retrying file operations that fail on Windows due to virus scanners

Going into the specifics using https://msdn.microsoft.com/en-us/library/t3ayayh1.aspx, it appears that errno 41 corresponds to

ENOTEMPTY: Directory not empty

which signals that the directory specified is not empty. It is an uncaught error because they don't have a branch that deals with this kind of error (ENOTEMPTY), while for example they have in the case it's another error like EPERM or EACCES.
In commit https://github.com/conda/conda/commit/fb2a783d9b9371559b5ea82aaf8ae631c2ce0450#diff-3757ed9862260ae3b54768b3e482e3fe
they explicitly remove reporting EPROTOTYPE as OSError(41, 'The directory is not empty'), so now you see that error number reported as a warning in the

log.warn("Uncaught backoff with errno %d", e.errno)

The other part they modify is in https://github.com/conda/conda/blob/fb2a783d9b9371559b5ea82aaf8ae631c2ce0450/conda/gateways/disk/delete.py, in the delete_trash() function, so now, if you would enable the info log, most probably you would see a line like

"Unable to fully clean trash directory %s\nThere are %d remaining file(s)."

enabled by

log.info("Unable to fully clean trash directory %s\nThere are %d remaining file(s).",
                 trash_dir, len(files_remaining))  

Now, delete_trash() is called by both your commands that you cite (install, update):
https://github.com/conda/conda/blob/f4b386476307e3979679957292d4f6e4c581df03/conda/cli/main_install.py
https://github.com/conda/conda/blob/a26b1eff17dcaf12f03aea5bbe8dee1e01308de7/conda/cli/main_update.py

As can be seen, delete_trash() is triggered respectively in 'install' and 'update' files previously mentioned respectively by the following code fragments:

from ..gateways.disk.delete import delete_trash

# some other code ...

def execute(args, parser):
    install(args, parser, 'install')
    delete_trash()

and

from ..gateways.disk.delete import delete_trash

# some other code ...

def execute(args, parser):
    install(args, parser, 'update')
    delete_trash()  

delete_trash() will then trigger that code path via backoff_rmdir() or backoff_unlink() that ultimately will lead to the warning you see from exp_backoff_fn() as seen before.
So, to sum up, the main chain of calls would be

update or install --> delete_trash() --> backoff_rmdir() or backoff_unlink() --> exp_backoff_fn() --> your warning message

According to the source code modifications done in the repository, the developers believe than that these are warning that you can safely ignore because these warnings happen in the "cleaning" phase of the update or installation commands, i.e. after update or install operation have been successfully performed.
I couldn't say 100% wether you could safely ignore these warnings. If after few attempts then the command to delete the trash directory succeeds then there's no problem. But if it doesn't succeed then you will hit the problem that this directory will grow bigger and bigger as the result of not deleting it. There were few issue opened in the repo for that and I don't know if the fixes cover the code path that you hit. My impression is that might not. To get further insights you could activate the info log level.

---

UPDATE: This issue https://github.com/conda/conda/issues/4164 exactly mention the warnings you reported, because people were getting long update and install time due to all the retries. As I mentioned that after all the retries (exponential back-offs) the delete operation could either succeed or fail, that person also mention this aspect in his report.
As you can see in here
https://github.com/conda/conda/issues/3664
there are few hacks that people employ to solve the problem of long waiting times because of the retries and will also make the next run of your command conda install X or conda update X without warning. These suggestions are:

1. (to speedup time for retries/delete) set MAX_TRIES = 1 in your copy of conda/gateways/disk/init.py
2. remove the .trash directory before you run the next conda install X or conda update X. See https://github.com/conda/conda/issues/3664 for a workaround employed by some people to automatize deletion of that directory using simple scripts. This should be usually safe to do it.

So the answers to your questions would be:
1) You can use the workaround mentioned in https://github.com/conda/conda/issues/3664, which is using the following powershell script (and another script):

$cir = conda info --root
$trash_dir = "$($cir)\pkgs\.trash"
if (Test-Path $trash_dir){
    Remove-Item -Recurse -Force $trash_dir
}
conda --debug update --all --yes --quiet 

to basically clean up that .trash directory;

2) You can safely ignore the warnings in the sense that they won't affect the functionality; the problem is that the more .trash gets populated, the more time and retries will go on to delete the items so you will hit performance problems; as you mentioned it's kind of having a "leak", but it doesn't affect the functionality. That directory is supposed to be emptied and removed, as it contains trash no longer needed. The system will try to delete it but might not be able to do it. So use 1).

---

UPDATE 2: As mentioned in one of my comments, one of the key changes is in the file conda/gateways/disk/__init__.py, which was a "fix" (https://github.com/conda/conda/commit/6cb3be39aec1c738678ae27b3a264941d08c859a) that made it to 4.3.6 version of conda (conda 4.3.6 release info) which solves the warning in question.
The key point to not seeing that warning is by having a branch that explicitly catches and deals with the kind of error explained before. Now, when en error of type ENOTEMPTY will happen (which is the error that was triggering the printing of the warning in this case), this will be caught and will not go to the branch that prints the warning examined by the question. To understand the main differences, in version 4.3.4 it was

elif e.errno in (ENOENT, EPROTOTYPE):                   
     raise
else:
     log.warn("Uncaught backoff with errno %d", e.errno)
     raise

while in version 4.3.6 it became:

elif e.errno in (ENOENT, ENOTEMPTY):    
    raise
else:
    log.warn("Uncaught backoff with errno %s %d", errorcode[e.errno], e.errno)
    raise

and you clearly see that now for that error will not go into the else branch so you won't see that message in this case.