Trying to read JSON file within a Python package

Question

I am in the process of packaging up a python package that I'll refer to as MyPackage.

The package structure is:

MyPackage/
    script.py
    data.json

The data.json file comprises cached data that is read in script.py.

I have figured out how to include data files (use of setuptools include_package_data=True and to also include path to data file in the MANIFEST.in file) but now when I pip install this package and import the installed MyPackage (currently testing install by pip from the GitHub repository) I get a FileNotFound exception (data.json) in the script that is to utilize MyPackage. However, I see that the data.json file is indeed installed in Lib/site-packages/MyPackage.

Am I doing something wrong here by trying to read in a json file in a package?

Note that in script.py I am attempting to read data.json as open('data.json', 'r')

Am I screwing up something regarding the path to the data file?

Answer 1

You're not screwing something up, accessing package resources is just a little tricky - largely because they can be packaged in formats where your .json might strictly speaking not exist as an actual file on the system where your package is installed (e.g. as zip-app). The right way to access your data file is not by specifying a path to it (like "MyPackage/data.json"), but by accessing it as a resource of your installed package (like "MyPackage.data.json"). The distinction might seem pedantic, but it can matter a lot.

Anyway, the access should be done using the builtin importlib.resources module:

import importlib.resources
import json

with importlib.resources.open_text("MyPackage", "data.json") as file:
    data = json.load(file)  
# you should be able to access 'data' like a dictionary here

---

If you happen to work on a python version lower than 3.7, you will have to install it as importlib_resources from pyPI.