Why is calling a Python lambda expression from C# not thread-safe?

Question

I define a side-effect-free (pure) lambda expression in IronPython and assign it to a C# delegate. When invoking the delegate simultaneously from multiple threads i get exceptions of type AccessViolationException, NullReferenceException and FatalEngineExecutionError.

The occurance of the error is non-deterministic and it mostly takes several million iterations to provoke it, which says "race condition" to me. How can i avoid it?

The exceptions are only raised when running the process with x64 (x86 does not crash) and outside of the debugger. The test system is a Core I7 (8 threads) on Windows 7, .NET Framework 4.0 and IronPython 2.7.1.

Here's the minimal code to produce the error:

var engine = Python.CreateEngine();  double a = 1.0; double b = 2.0;  while (true) {     Func<double, double, double> calculate = engine.Execute("lambda a,b : a+b");      System.Threading.Tasks.Parallel.For(0, 1000, _ =>     {          for (int i = 0; i < 1000; i++) { calculate(a,b); }     });      Console.Write(".");    } 

Error message:

FatalExecutionEngineError was detected

Message: The runtime has encountered a fatal error. The address of the error was at 0xf807829e, on thread 0x3da0. The error code is 0xc0000005. This error may be a bug in the CLR or in the unsafe or non-verifiable portions of user code. Common sources of this bug include user marshaling errors for COM-interop or PInvoke, which may corrupt the stack.

Update: Even if the engine is declared as thread-local, it crashes after some time:

var calculate = new ThreadLocal<Func<double, double, double>>(() => Python.CreateEngine().Execute("lambda a,b : a+b")); 

Answer 1

This is likely due to a race condition within the ScriptEngine. Note that ScriptEngine.Execute returns a reference to a PythonFunction rather than a Func (it's due to C#'s dynamic behavior, that you can treat the result as a Func. I'm no expert on IronPython but looking at the source of PythonFunction, there is no indication whatsoever that it is threadsafe.

Answer 2

Have you tried replacing the 64 bit data type (double) with a 32 bit data type (float)? I do know that "A simple read or write on a field of 32 bits or less is always atomic" whereas a 64 bit field can have problems depending on the processor and the code. It seems like kind of a long shot but it is worth a try.