Use pathlib for S3 paths

Question

I would like to build some functionality to move files between S3 and my local file system, but pathlib appears to combine repeated slashes, breaking my aws-cli functionality:

>>> from pathlib import Path  >>> str(Path('s3://loc')) s3:/loc' 

How can I manipulate S3 paths in this way?

Answer 1

Using s3path package

The s3path package makes working with S3 paths a little less painful. It is installable from PyPI or conda-forge. Use the S3Path class for actual objects in S3 and otherwise use PureS3Path which shouldn't actually access S3.

Although the previous answer by metaperture did mention this package, it didn't include the URI syntax.

Also be aware that this package has certain deficiencies which are reported in its issues.

>>> from s3path import PureS3Path  >>> PureS3Path.from_uri('s3://mybucket/foo/bar') / 'add/me' PureS3Path('/mybucket/foo/bar/add/me')  >>> _.as_uri() 's3://mybucket/foo/bar/add/me' 

Note the instance relationships to pathlib:

>>> from pathlib import Path, PurePath >>> from s3path import S3Path, PureS3Path  >>> isinstance(S3Path('/my-bucket/some/prefix'), Path) True >>> isinstance(PureS3Path('/my-bucket/some/prefix'), PurePath) True 

Using pathlib.Path

This is a lazier version of the answer by kichik using only pathlib. This approach is not necessarily recommended. It's just not always entirely necessary to use urllib.parse.

>>> from pathlib import Path  >>> orig_s3_path = 's3://mybucket/foo/bar' >>> orig_path = Path(*Path(orig_s3_path).parts[1:]) >>> orig_path PosixPath('mybucket/foo/bar')  >>> new_path = orig_path / 'add/me' >>> new_s3_path = 's3://' + str(new_path) >>> new_s3_path 's3://mybucket/foo/bar/add/me' 

Using os.path.join

For simple joins only, how about os.path.join?

>>> import os  >>> os.path.join('s3://mybucket/foo/bar', 'add/me') 's3://mybucket/foo/bar/add/me' >>> os.path.join('s3://mybucket/foo/bar/', 'add/me') 's3://mybucket/foo/bar/add/me' 

os.path.normpath cannot however be naively used:

>>> os.path.normpath('s3://mybucket/foo/bar')  # Converts 's3://' to 's3:/' 's3:/mybucket/foo/bar'