Same Python code, same data, different results on different machines

Question

I have a very strange problem that I get different results on the same code and same data on different machines.

I have a python code based on numpy/scipy/sklearn and I use anaconda as my base python distribution. Even when I copy the entire project directory (which includes all the data and code) from my main machine to another machine and run it, the results I get are different. Specifically, I'm doing a classification task and I get 3 percent difference in accuracy. I am using the same version of python and anaconda on the two machines. My main machine is ubuntu 16.04 and the results on it are lower than several other machines with various OS on which I tried (OSX, ubuntu 14.04 and Centos). So, there should be something wrong with my current system configuration because all other machines show consistent results. Since the version of my anaconda is consistent among all machines, I have no idea what else could be the problem. Any ideas what else I should check or what could be the source of the problem?

I also removed and reinstalled anaconda from scratch but it didn't help.

Answer 1

I had a similar problem and I found this discussion. May be the problem is that MKL(Intel Math Kernel Library) float point operations are non-deterministic by default. So export MKL_CBWR=AUTO may solve the problem.

Answer 2

If your code uses linear algebra, check it. Generally, roundoff errors are not deterministic, and if you have badly conditioned matrices, it can be it.