Enqueuing Instance Methods or Objects in RQ?

Question

So, RQ explicitly states I can enqueue an instance method of an object here, so I've been trying to do that, but getting a PicklingError:

q.enqueue(some_obj.some_func, some_data)
*** PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

Really, I just need access to a SQL connection in my method, so I tried just making it a function that takes the SQL connection explicitly. That fails too:

q.enqueue(some_func, sql_sess, some_data)
*** PicklingError: Can't pickle <class 'sqlalchemy.orm.session.Session'>: it's not the same object as sqlalchemy.orm.session.Session

How do I get around this? Am I doing something wrong, or is the library just broken?

Answer 1

When enqueueing a job, the pickling that gets done is attempted on your input to your blocking function.

Similar to you, I originally had code that passed in a SQLAlchemy object into my enqueued function. This code was in a file called init.py.

user_emails = session.query(UserEmail).all()
for sql_alchemy_email_object in emails:
    q.enqueue(parse_email_function, sql_alchemy_email_object)

Which produced the PicklingError. What I did to resolve this was do the individual lookup for the SQLAlchemy mapped email_object in the parse_email_function by a single ID.

So I then changed the code to this in init.py:

user_email_ids = [email.id for email in session.query(UserEmail).all()]
for email_id in user_email_ids:
    q.enqueue(parse_email_function, email_id)

Then, in the file containing parse_email_function, I grabbed the individual Email Object from SQLAlchemy by the ID I had passed into the parse_email_function().

Since you asked about a way to get around this, one way to avoid this is by simply avoiding passing in the Object directly to your processing function, and instead pass in a static id to perform the lookup on in your processing function code.