Package and find non Python Files in a python package

Question

I'm fairly new to python packaging and I'm trying to create a command line tool so that I can send to client to interact with my service in AWS.

My goal is to have a command line tool to upload files that are in the folder resources to s3 that will later be used by other services.

It's my first time using setuptools for that but I'm seem to be lost at some point.

My project structure is something like:

ProjectRoot
├── MANIFEST.in
├── Pipfile
├── Pipfile.lock
├── dist
│   ├── myscript-0.0.1.whl
│   └── myscript-0.0.1.tar.gz
├── pyproject.toml
├── resources
│   ├── artifacts
│   │   ├── code1.jar
│   │   ├── code2.jar
│   │   ├── api.keys
│   │   ├── package1.tar.gz
│   │   ├── install-linux.sh
│   │   └── confs.yaml
│   ├── recipe.template.yaml
└── src
    └── code
        ├── __init__.py
        └── myscript.py

I've tried to make setuptools add the files to the .tar package with the pyproject.toml with this:

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "myscript"
version = "0.0.1"
dependencies = [
    'Click',
    'boto3',
    'botocore',
]

[project.scripts]
myscript = "code.main:run"

[tool.setuptools]
include-package-data = true

[tool.setuptools.packages.find]
where = ["src","resources"] 
include = ["code*"]
exclude = [] 

[tool.setuptools.package-data]
"resources.artifacts" = ["*"]
recipe = ["*.yaml"]

After that I try to install the wheel generated file with pip install dist/generated_file.whl, but I can't find the resources/ folder anywhere during installation.

ps.: I also got a little lost if I need the whl and the tar package together.

I tried using relative paths to find the resources, but I saw they weren't installed in the sites_packages.

My latest try was using from importlib_resources import files but it also can't seem to find the resources.

I can't find the resources folder files.

Answer 1

Starting point

With the given project structure

📁 <project root>/
├─📄 pyproject.toml
├─📁 src/
│ └─📁 code/
│   ├─📄 __init__.py
│   └─📄 myscript.py
├─📁 resources/
  └─📁 artifacts/
    └─📄 code1.jar

and by specifying

[tool.setuptools.packages.find]
where = ["src","resources"]
include = ["code*"]
exclude = []

[tool.setuptools.package-data]
"resources.artifacts" = ["*"]
recipe = ["*.yaml"]

you'll get a wheel with following contents

📁 myscript-0.0.1-py3-none-any/
└─📁 code/
  ├─📄 __init__.py
  └─📄 myscript.py

The reason for this is that the only package found is code. A package is a folder with python (.py) files, and usually with __init__.py file (if not talking about namespace packages which are a bit of a special thing).

What I would do?

• First, renaming your main package folder. You've called your project myscript in pyproject (so you would install it with pip install myscript), but then the file structure would imply that the import name is code; so you would need to do import code.myscript (code being the main package). I'll change the project name to be in this example myproj, by changing the name in pyproject.toml and ./code folder to ./myproj

• Second, the name of the "package-data" to me says that it is data inside a package. The ./resources is not a package as it does not contain any python files. If you add there an empty __init__.py, it will become a package. But there is another problem in pyproject.toml: your package should be found under a folder called ./resources, but that is actually in root folder (.). Therefore, you should either change where = ['src', '.'] (which creates new problems) or move ./resources to ./resources/resources, but you could also do it easier (third point)

• Third, you could simplify things by putting the data files inside you package (./myproj). That's fair more common practice, and also makes pip install the resources with your code, inside site-packages/myproj, which is nice (although, there are other possibilities). So, I propose these changes to pyproject.toml:

[project]
name = "myproj" # <-- this changed
version = "0.0.1"

[tool.setuptools.packages.find]
where = ["src"] # <-- this changed

[tool.setuptools.package-data]
"*" = ["*.*"] # <-- this changed

and then the folder structure to

📁 <project root>/
├─📄 pyproject.toml
└─📁 src/
  └─📁 myproj/
    ├─📄 __init__.py
    ├─📁 __assets__/ # non-code files here
    │ └─📄 code1.jar     
    └─📄 myscript.py

That will then create a wheel with following folder structure:

└─📁 myproj-0.0.1-py3-none-any/
  └─📁 myproj/
    ├─📄 __init__.py
    ├─📁 __assets__/
    │ └─📄 code1.jar
    └─📄 myscript.py