Import failure of s3fs library in AWS Glue

Question

AWS glue is not importing s3fs module

import s3fs

I expect the library to be imported but AWS glue says

ImportError : No module named s3fs

Answer 1

AWS Glue jobs come with some common libraries pre installed but for anything more than that you need to download the .whl for the library from pypi, which in the case of s3fs can be found here.

Once you have that, upload it to an s3 bucket, eg. s3://my-libraries/ and reference it in the Python library path field in the console.

This will prompt Glue to install the libraries within this bucket prior to running the script. Note that only pure python libraries are support currently.