aiobotocore - ImportError: cannot import name 'InvalidIMDSEndpointError'

Question

The code below raise an import exception

import s3fs
fs = s3fs.S3FileSystem(anon=False)

Exception

Traceback (most recent call last):
  File "issue.py", line 1, in <module>
    import s3fs
  File "/home/ubuntu/.local/lib/python3.6/site-packages/s3fs/__init__.py", line 1, in <module>
    from .core import S3FileSystem, S3File
  File "/home/ubuntu/.local/lib/python3.6/site-packages/s3fs/core.py", line 14, in <module>
    import aiobotocore
  File "/home/ubuntu/.local/lib/python3.6/site-packages/aiobotocore/__init__.py", line 1, in <module>
    from .session import get_session, AioSession
  File "/home/ubuntu/.local/lib/python3.6/site-packages/aiobotocore/session.py", line 6, in <module>
    from .client import AioClientCreator, AioBaseClient
  File "/home/ubuntu/.local/lib/python3.6/site-packages/aiobotocore/client.py", line 12, in <module>
    from .utils import AioS3RegionRedirector
  File "/home/ubuntu/.local/lib/python3.6/site-packages/aiobotocore/utils.py", line 10, in <module>
    from botocore.exceptions import (
ImportError: cannot import name 'InvalidIMDSEndpointError'

More details:
OS: Ubuntu 18.04
Python version 3.6

pip3 show s3fs

Name: s3fs
Version: 0.5.1
Summary: Convenient Filesystem interface over S3
Home-page: http://github.com/dask/s3fs/
Location: /home/ubuntu/.local/lib/python3.6/site-packages
Requires: fsspec, aiobotocore

 pip3 show botocore

Name: botocore
Version: 1.19.24
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Location: /home/ubuntu/.local/lib/python3.6/site-packages Requires: jmespath, urllib3, python-dateutil

  pip3 show aiobotocore

Name: aiobotocore
Version: 1.2.0
Summary: Async client for aws services using botocore and aiohttp
Home-page: https://github.com/aio-libs/aiobotocore
Location: /home/ubuntu/.local/lib/python3.6/site-packages
Requires: botocore, wrapt, aioitertools, aiohttp

 pip3 show boto3

Name: boto3
Version: 1.16.52
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Location: /home/ubuntu/.local/lib/python3.6/site-packages
Requires: botocore, s3transfer, jmespath

---

any ideas what is wrong ?

Thanks

Answer 1

You can also downgrade aiobotocore to the older version 1.1.2. S3fs==0.5.1 only depends on aiobotocore >= 1.0.1. Use johnnydep to visualize the dependency tree (pip install johnnydep)

$ johnnydep s3fs==0.5.1

s3fs==0.5.1                              Convenient Filesystem interface over S3
├── aiobotocore>=1.0.1                   Async client for aws services using botocore and aiohttp
│   ├── aiohttp>=3.3.1                   Async http client/server framework (asyncio)
│   │   ├── async-timeout<4.0,>=3.0      Timeout context manager for asyncio programs
│   │   ├── attrs>=17.3.0                Classes Without Boilerplate
│   │   ├── chardet<4.0,>=2.0            Universal encoding detector for Python 2 and 3
│   │   ├── multidict<7.0,>=4.5          multidict implementation
│   │   ├── typing-extensions>=3.6.5     Backported and Experimental Type Hints for Python 3.5+
│   │   └── yarl<2.0,>=1.0               Yet another URL library
│   │       ├── idna>=2.0                Internationalized Domain Names in Applications (IDNA)
│   │       └── multidict>=4.0           multidict implementation
│   ├── aioitertools>=0.5.1              itertools and builtins for AsyncIO and mixed iterables
│   │   └── typing_extensions>=3.7       Backported and Experimental Type Hints for Python 3.5+
│   ├── botocore<1.19.53,>=1.19.52       Low-level, data-driven core of boto 3.
│   │   ├── jmespath<1.0.0,>=0.7.1       JSON Matching Expressions
│   │   ├── python-dateutil<3.0.0,>=2.1  Extensions to the standard Python datetime module
│   │   │   └── six>=1.5                 Python 2 and 3 compatibility utilities
│   │   └── urllib3<1.27,>=1.25.4        HTTP library with thread-safe connection pooling, file post, and more.
│   └── wrapt>=1.10.10                   Module for decorators, wrappers and monkey patching.
└── fsspec>=0.8.0                        File-system specification