Python 3 Best practices on distinguish production and development configuration

Question

I'm working on an embedded system project where my dev setup is different than my prod. The differences include variables and packages imports.

What is the best way to structure the config files for python3 application where dev and prod setups are different?

prod: My device exchange messages (using pyserial) with an electronic system and also communicates with a server.

dev: I use a fake and fixed response from a function to mock both the electronic and server responses. Even if the functions that I mock are essential in prod they are less in dev. I can mock them because the most important part of this project are the functions that use and treat them. So, there are packages imports and function calls that do not make sense and introduce errors in dev mode.

Every time I need to switch from one to another I need to change a good amount of the code and some times there are errors introduced. I know this is really (💩) not the best approach and I wanted to know what are the best practices.

The Closest Solution Found

Here there is a good solution to set up different variables for each environment. I was hoping for something similar but for projects that require different packages import for different environments.

My Setup

Basic workflow:

• A task thread is executed each second
  • module_1 do work and call module_2
  • module_2 do work and call module_3
  • module_3 do work and send back a response

Basic folder structure:

• root
  • main
  • config.py
  • /config
    • prod
    • dev
  • /mod_1
  • /mod_2
  • /mod_3
  • /replace_imports

module_1 and module_3 use, each one, a specific package for prod and must be replaced by a dev function

What do I have:

# config.py
if os.environ["app"] == "dev":
    import * from root.config.dev
if os.environ["app"] == "prod":
    import * from root.config.prod

# config/prod.py
import _3rd_party_module_alpha
import _3rd_party_module_beta
...
obj_alpha = _3rd_party_module_alpha()
func_beta = _3rd_party_module_beta()

# config/dev.py
import * from root.replace_imports

# replace_imports.py
obj_alpha = fake_3rd_party_module_alpha()
func_beta = fake_3rd_party_module_beta()

Answer 1

You really should not have code changes between a dev at point X, and pushing into QA/CI , then prod at point X. Your dev and prod code can be expected to be different at different stages, of course, and version control is key. But moving to production should not require code changes, just config changes.

Environment variables (see 12 factor app stuff) can help, but sometimes config is in code, for example in Django setting files.

In environments like Django where "it points to" a settings file, I've seen this kinda of stuff:

base_settings.py:

common config

dev_settings.py:

#top of file
import * from base_settings


... dev specifics, including overrides of base...

edit: I am well aware of issues with import *. First, this is a special case, for configurations, where you want to import everything. Second, the real issue with import * is that it clobbers the current namespace. That import is right at the top so that won't happen. Linters aside, and they can be suppressed for just that line, the leftover issue is that you may not always know where a variable magically came from, unless you look in base.

prod_settings.py:

import * from base_settings

...production specifics, including overrides of base...

Advanced users of webpack configuration files (and those are brutal) do the same thing, i.e. use a base.js then import that into a dev.js and prod.js.

The key here is to have as much as possible in base, possibly with the help of environment variables (be careful not to over-rely on those, no ones likes apps with dozens of environment variable settings). Then dev and prod are basically about keys, paths, urls, ports, that kind of stuff. Make sure to keep secrets out of them however, because they gravitate there naturally but have no business being under version control.

re. your current code

appname = os.getenv("app")

if appname == "dev":

    #outside of base=>dev/prod merges, at top of "in-config" namespaces, 
    #I would avoid `import *` here and elsewhere

    import root.config.dev as config

elif appname == "prod":

    import root.config.prod as config

else:
    raise ValueError(f"environment variable $app should have been either `dev` or `prod`. got:{appname}:")

Last, if you are stuck without an "it points to Python settings file" mechanism, similar to that found in Django, you can (probably) roll your own by storing the config module path (xxx.prod,xxx.dev) in an environment variable and then using a dynamic import. Mind you, your current code largely does that already except that you can't experiment with/add other settings files.

Don't worry if you don't get it right right away, and don't over-design up front - it takes a while to find what works best for you/your app.

Answer 2

Pipenv was specially created for these things and more