Using different versions of a python library in the same process

Question

We've got a python library that we're developing. During development, I'd like to use some parts of that library in testing the newer versions of it. That is, use the stable code in order to test the development code. Is there any way of doing this in python?

Edit: To be more specific, we've got a library (LibA) that has many useful things. Also, we've got a testing library that uses LibA in order to provide some testing facilities (LibT). We want to test LibA using LibT, but because LibT depends on LibA, we'd rather it to use a stable version of LibA, while testing LibT (because we will change LibT to work with newer LibA only once tests pass etc.). So, when running unit-tests, LibA-dev tests will use LibT code that depends on LibA-stable.

One idea we've come up with is calling the stable code using RPyC on a different process, but it's tricky to implement in an air-tight way (making sure it dies properly etc, and allowing multiple instances to execute at the same time on the same computer etc.).

Thanks

Answer 1

"We want to test LibA using LibT, but because LibT depends on LibA, we'd rather it to use a stable version of LibA, while testing LibT "

It doesn't make sense to use T + A to test A. What does make sense is the following.

LibA is really two things mashed together: A1 and A2.

T depends on A1.

What's really happening is that you're upgrading and testing A2, using T and A1.

If you decompose LibA into the parts that T requires and the other parts, you may be able to break this circular dependency.

Answer 2

If you "test" libA-dev using libT which depends on libA (stable), then you are not really testing libA-dev as it would behave in a production environment. The only way to really test libA-dev is to take the full plunge and make libT depend on libA-dev. If this breaks your unit tests then that is a good thing -- it is showing you what needs to be fixed.

If you don't have unit tests, then this is the time to start making them (using stable libA and libT first!).

I recommend using a "version control system" (e.g. bzr,hg,svn,git). Then you could make branches of your project, "stable" and "devA".

To work on branch devA, you would first run

export PYTHONPATH=/path/to/devA

By making sure the PYTHONPATH environment variable excludes the other branches, you're assured Python is using just the modules you desire.

When the time comes to merge code from dev --> stable, the version control software will provide an easy way to do that too.

Version control also allows you to be bolder -- trying major changes is not as scary. If things do not work out, reverting is super easy. Between that and the PYTHONPATH trick, you are always able to return to known, working code.

If you feel the above just simply is not going to work for you, and you must use libT-which-depends-on-libA to test libA-dev, then you'll need to rename all the modules and modify all the import statements to make a clear separation between libA-dev and libA. For example, if libA has a module called moduleA.py, then rename it moduleA_dev.py.

The command

rename -n 's/^(.*)\.py/$1_dev.py/' *.py

will add "_dev" to all the *.py files. (With the "-n" flag the rename command will only show you the contemplated renaming. Remove the "-n" to actually go through with it.)

To revert the renaming, run

rename -n 's/^(.*)_dev\.py/$1.py/' *.py

Next you'll need to change all references to moduleA to moduleA_dev within the code. The command

find /path/to/LibA-dev/ -type f -name '*.py' -exec sed -i 's/moduleA/moduleA_dev/g' {} \;

will alter every *.py file in LibA-dev, changing "moduleA" --> "moduleA_dev".

Be careful with this command. It is dangerous, because if you have a variable called moduleAB then it will get renamed moduleA_devB, while what you really wanted might be moduleAB_dev.

To revert this change (subject to the above caveat),

find /path/to/LibA-dev/ -type f -name '*.py' -exec sed -i 's/moduleA_dev/moduleA/g' {} \;

Once you separate the namespaces, you've broken the circular dependency. Once you are satisfied your libA-dev is good, you could change moduleA_dev.py --> moduleA.py and changes all references to moduleA_dev --> moduleA in your code.