TypeError: cannot pickle '_thread.lock' object with RQ

Question

I am using RQ with flask for queuing jobs in loop. I have the following code :

from rq import Queue
from rq.job import Job
from worker import conn

q = Queue(connection=conn)

for i in range(5):
    job = q.enqueue_call(
    func=process_data, args=(i, data,))
    print(job.get_id()) 

Now I am getting the error :

TypeError: cannot pickle '_thread.lock' object

I have the worker with following code :

import os

import redis
from rq import Worker, Queue, Connection

listen = ['default']

redis_url = os.getenv('REDISTOGO_URL', 'redis://localhost:6379')

conn = redis.from_url(redis_url)

if __name__ == '__main__':
    with Connection(conn):
        worker = Worker(list(map(Queue, listen)))
        worker.work()

How this can be corrected ?

Answer 1

I solved a similar problem by downgrading from Python 3.8 to Python 3.7

My situation was a little different. I am running a Django server, which schedules tasks using Django-Q. However, Django-Q is based on RQ, and the error

TypeError: cannot pickle '_thread.lock' object

is thrown by Python's Multiprocessing module, so I believe the solution will translate.

As of May 2020, I expect that this is a bug, although it is unclear what causes it.