SqlAlchemy and Multiprocessing

Question

I use SqlAlchemy to connect to my database backend and make heavy use of multiprocessing in my Python application. I came to a situation which requires to pass an object reference, which is the result of a database query, from one process to another.

This is a problem, because when accessing an attribute of the object, SqlAlchemy trys to reattach the object into the current session of the other process, which fails with an exception, because the object is attached in an other session:

InvalidRequestError: Object '<Field at 0x9af3e4c>' is already attached to session '148848780' (this is '159831148')

What is the way to handle this situation? Is it possible to detach the object from the first session or clone the object without the ORM related stuff?

Answer 1

This is a bad idea (tm).

You shouldn't share a stateful object between processes like this (I know it's tempting) because all kinds of bad things can happen since lock primitives are not intended to work across multiple python runtimes.

I suggest taking the attributes you need out of that object, jamming them into a dict and sending it across processes using multprocessing Pipes:

http://docs.python.org/library/multiprocessing.html#pipes-and-queues